Jul 17 21:17:02.722577 ip-10-0-135-220 systemd[1]: Starting Kubernetes Kubelet...
Jul 17 21:17:03.123276 ip-10-0-135-220 kubenswrapper[2546]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:17:03.123276 ip-10-0-135-220 kubenswrapper[2546]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 17 21:17:03.123816 ip-10-0-135-220 kubenswrapper[2546]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:17:03.123816 ip-10-0-135-220 kubenswrapper[2546]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 17 21:17:03.123816 ip-10-0-135-220 kubenswrapper[2546]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 17 21:17:03.124253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.124171    2546 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 17 21:17:03.131636 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131614    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.131636 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131633    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.131636 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131637    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.131636 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131639    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.131636 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131642    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131645    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131648    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131651    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131653    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131656    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131658    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131661    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131663    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131666    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131668    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131672    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131676    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131679    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131682    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131684    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131687    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131689    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131692    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.131821 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131694    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131697    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131700    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131702    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131704    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131707    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131710    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131713    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131715    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131718    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131720    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131724    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131726    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131729    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131731    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131736    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131738    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131741    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131744    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131746    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.132281 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131749    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131751    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131754    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131756    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131758    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131761    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131765    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131769    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131772    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131775    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131777    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131780    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131783    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131785    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131788    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131790    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131793    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131795    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131798    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.132772 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131800    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131803    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131806    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131808    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131811    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131813    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131816    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131818    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131821    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131824    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131826    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131829    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131832    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131834    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131837    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131840    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131842    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131845    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131847    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131849    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.133220 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131852    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131855    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131857    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131860    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131862    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131864    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131867    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131871    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131875    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131878    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131881    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131883    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131886    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131888    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131891    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131893    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.131895    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132608    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132616    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132619    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.133718 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132623    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132626    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132629    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132631    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132634    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132638    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132642    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132645    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132648    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132651    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132653    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132656    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132658    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132661    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132664    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132666    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132669    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132671    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132674    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.134186 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132676    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132679    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132681    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132683    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132686    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132688    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132691    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132693    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132696    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132698    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132700    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132703    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132706    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132709    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132711    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132714    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132716    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132719    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132721    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.134668 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132724    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132726    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132728    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132732    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132736    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132739    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132741    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132744    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132746    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132749    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132752    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132755    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132757    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132759    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132762    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132764    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132767    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132769    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132772    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.135125 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132774    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132777    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132780    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132782    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132785    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132790    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132793    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132795    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132798    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132800    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132803    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132806    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132808    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132810    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132813    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132815    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132818    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132821    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132823    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132826    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.135592 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132828    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132831    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132833    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132836    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132840    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132843    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132846    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132848    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132852    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132855    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132858    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132860    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132863    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132866    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132868    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132870    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132873    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132879    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.132881    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133262    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.136068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133267    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133270    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133273    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133275    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133278    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133281    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133284    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133287    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133311    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133314    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133317    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133320    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133322    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133325    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133327    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133330    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133333    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133336    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133339    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133341    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.136596 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133344    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133346    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133349    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133351    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133353    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133359    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133363    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133366    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133368    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133372    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133374    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133377    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133380    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133383    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133385    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133388    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133390    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133393    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133395    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133398    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.137072 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133400    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133403    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133407    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133409    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133412    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133414    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133416    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133421    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133425    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133428    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133430    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133433    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133435    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133438    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133441    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133443    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133445    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133449    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133452    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.137569 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133454    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133456    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133461    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133463    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133466    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133469    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133472    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133475    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133477    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133480    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133483    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133485    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133487    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133490    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133492    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133495    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133497    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133500    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133502    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133505    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.138056 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133507    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133509    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133514    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133517    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133520    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133522    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133525    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133527    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133529    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133532    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133536    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133538    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133541    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133543    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133548    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133551    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133553    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133556    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.133558    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133630    2546 flags.go:64] FLAG: --address="0.0.0.0"
Jul 17 21:17:03.138557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133638    2546 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133644    2546 flags.go:64] FLAG: --anonymous-auth="true"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133648    2546 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133652    2546 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133655    2546 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133659    2546 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133663    2546 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133666    2546 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133669    2546 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133672    2546 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133675    2546 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133678    2546 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133681    2546 flags.go:64] FLAG: --cgroup-root=""
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133683    2546 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133686    2546 flags.go:64] FLAG: --client-ca-file=""
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133690    2546 flags.go:64] FLAG: --cloud-provider="external"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133693    2546 flags.go:64] FLAG: --cluster-dns="[]"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133697    2546 flags.go:64] FLAG: --cluster-domain=""
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133700    2546 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133703    2546 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133706    2546 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133709    2546 flags.go:64] FLAG: --container-log-max-files="5"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133714    2546 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 17 21:17:03.139042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133717    2546 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133720    2546 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133723    2546 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133728    2546 flags.go:64] FLAG: --contention-profiling="false"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133731    2546 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133734    2546 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133737    2546 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133739    2546 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133743    2546 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133746    2546 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133749    2546 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133751    2546 flags.go:64] FLAG: --enable-load-reader="false"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133754    2546 flags.go:64] FLAG: --enable-server="true"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133757    2546 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133761    2546 flags.go:64] FLAG: --event-burst="100"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133764    2546 flags.go:64] FLAG: --event-qps="50"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133767    2546 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133770    2546 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133773    2546 flags.go:64] FLAG: --eviction-hard=""
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133776    2546 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133779    2546 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133781    2546 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133784    2546 flags.go:64] FLAG: --eviction-soft=""
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133787    2546 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133790    2546 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 17 21:17:03.139637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133793    2546 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133796    2546 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133799    2546 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133801    2546 flags.go:64] FLAG: --fail-swap-on="true"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133804    2546 flags.go:64] FLAG: --feature-gates=""
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133807    2546 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133811    2546 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133814    2546 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133817    2546 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133820    2546 flags.go:64] FLAG: --healthz-port="10248"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133823    2546 flags.go:64] FLAG: --help="false"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133826    2546 flags.go:64] FLAG: --hostname-override="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133829    2546 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133832    2546 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133834    2546 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133838    2546 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133841    2546 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133843    2546 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133846    2546 flags.go:64] FLAG: --image-service-endpoint=""
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133849    2546 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133851    2546 flags.go:64] FLAG: --kube-api-burst="100"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133854    2546 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133857    2546 flags.go:64] FLAG: --kube-api-qps="50"
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133860    2546 flags.go:64] FLAG: --kube-reserved=""
Jul 17 21:17:03.140240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133862    2546 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133865    2546 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133868    2546 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133871    2546 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133873    2546 flags.go:64] FLAG: --lock-file=""
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133876    2546 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133878    2546 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133881    2546 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133886    2546 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133889    2546 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133891    2546 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133894    2546 flags.go:64] FLAG: --logging-format="text"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133897    2546 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133900    2546 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133904    2546 flags.go:64] FLAG: --manifest-url=""
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133906    2546 flags.go:64] FLAG: --manifest-url-header=""
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133910    2546 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133913    2546 flags.go:64] FLAG: --max-open-files="1000000"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133916    2546 flags.go:64] FLAG: --max-pods="110"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133919    2546 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133922    2546 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133925    2546 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133928    2546 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133931    2546 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133933    2546 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 17 21:17:03.140851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133937    2546 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133943    2546 flags.go:64] FLAG: --node-status-max-images="50"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133946    2546 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133949    2546 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133952    2546 flags.go:64] FLAG: --pod-cidr=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133955    2546 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133960    2546 flags.go:64] FLAG: --pod-manifest-path=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133963    2546 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133966    2546 flags.go:64] FLAG: --pods-per-core="0"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133968    2546 flags.go:64] FLAG: --port="10250"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133971    2546 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133974    2546 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0dcdc4800076fe416"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133977    2546 flags.go:64] FLAG: --qos-reserved=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133980    2546 flags.go:64] FLAG: --read-only-port="10255"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133983    2546 flags.go:64] FLAG: --register-node="true"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133986    2546 flags.go:64] FLAG: --register-with-taints=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133989    2546 flags.go:64] FLAG: --registry-burst="10"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133992    2546 flags.go:64] FLAG: --registry-qps="5"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133995    2546 flags.go:64] FLAG: --reserved-cpus=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.133998    2546 flags.go:64] FLAG: --reserved-memory=""
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134001    2546 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134005    2546 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134008    2546 flags.go:64] FLAG: --rotate-certificates="false"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134011    2546 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134014    2546 flags.go:64] FLAG: --runonce="false"
Jul 17 21:17:03.141464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134016    2546 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134019    2546 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134022    2546 flags.go:64] FLAG: --seccomp-default="false"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134025    2546 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134028    2546 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134030    2546 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134033    2546 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134036    2546 flags.go:64] FLAG: --storage-driver-password="root"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134039    2546 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134041    2546 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134044    2546 flags.go:64] FLAG: --storage-driver-user="root"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134047    2546 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134049    2546 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134052    2546 flags.go:64] FLAG: --system-cgroups=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134055    2546 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134060    2546 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134062    2546 flags.go:64] FLAG: --tls-cert-file=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134065    2546 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134069    2546 flags.go:64] FLAG: --tls-min-version=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134072    2546 flags.go:64] FLAG: --tls-private-key-file=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134074    2546 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134077    2546 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134080    2546 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134082    2546 flags.go:64] FLAG: --v="2"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134086    2546 flags.go:64] FLAG: --version="false"
Jul 17 21:17:03.142069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134090    2546 flags.go:64] FLAG: --vmodule=""
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134093    2546 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.134097    2546 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134209    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134212    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134216    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134218    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134221    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134224    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134226    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134229    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134231    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134234    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134237    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134239    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134241    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134244    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134246    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134249    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134251    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.142839 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134253    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134256    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134258    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134260    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134263    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134265    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134268    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134270    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134272    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134275    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134278    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134280    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134283    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134286    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134304    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134311    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134314    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134316    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134320    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.143338 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134324    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134327    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134331    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134334    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134336    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134339    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134342    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134344    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134347    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134349    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134352    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134354    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134357    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134359    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134362    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134364    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134367    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134369    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134371    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.143808 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134374    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134376    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134379    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134381    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134384    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134386    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134389    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134391    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134394    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134398    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134400    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134403    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134407    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134410    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134413    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134417    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134420    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134423    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134425    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134428    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.144277 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134431    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134434    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134437    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134440    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134442    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134445    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134447    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134450    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134452    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134455    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134457    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134460    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134462    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134465    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134467    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134470    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134472    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134475    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134477    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134479    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.144779 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134482    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134487    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134490    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.134492    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.135269    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.141835    2546 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.141849    2546 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.142653    2546 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.142660    2546 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142717    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142722    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142725    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142728    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142731    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142734    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.145251 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142737    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142740    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142743    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142745    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142748    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142750    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142753    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142755    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142758    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142760    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142763    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142765    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142768    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142770    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142772    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142775    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142777    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142780    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142783    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142785    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.145684 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142789    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142791    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142794    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142797    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142800    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142802    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142805    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142807    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142809    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142812    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142814    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142817    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142819    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142822    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142824    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142827    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142829    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142831    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142834    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142836    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.146170 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142838    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142841    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142844    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142846    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142849    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142852    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142856    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142860    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142863    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142865    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142868    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142871    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142873    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142876    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142879    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142882    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142884    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142887    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142889    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142892    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.146698 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142894    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142896    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142899    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142902    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142904    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142906    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142909    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142911    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142913    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142917    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142921    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142924    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142926    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142929    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142932    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142936    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142939    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142941    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142944    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.147176 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142947    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142949    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142952    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142954    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142956    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142959    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142962    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142965    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142968    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142970    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142972    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142975    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142977    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.142979    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.147669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.142985    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143089    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143093    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143096    2546 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143099    2546 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143101    2546 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143104    2546 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143107    2546 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143109    2546 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143112    2546 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143114    2546 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143117    2546 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143119    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143121    2546 feature_gate.go:328] unrecognized feature gate: Example
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143124    2546 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143126    2546 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143128    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143132    2546 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143136    2546 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143138    2546 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 17 21:17:03.148017 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143141    2546 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143144    2546 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143148    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143150    2546 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143153    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143156    2546 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143159    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143162    2546 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143164    2546 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143167    2546 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143169    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143171    2546 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143174    2546 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143176    2546 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143179    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143181    2546 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143183    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143186    2546 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143188    2546 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143191    2546 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 17 21:17:03.148515 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143193    2546 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143196    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143198    2546 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143201    2546 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143203    2546 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143205    2546 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143208    2546 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143210    2546 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143212    2546 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143215    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143217    2546 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143219    2546 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143222    2546 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143224    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143227    2546 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143229    2546 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143232    2546 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143235    2546 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143237    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 17 21:17:03.148981 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143240    2546 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143244    2546 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143247    2546 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143249    2546 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143252    2546 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143255    2546 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143258    2546 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143260    2546 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143263    2546 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143265    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143268    2546 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143270    2546 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143272    2546 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143275    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143277    2546 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143280    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143282    2546 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143285    2546 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143287    2546 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 17 21:17:03.149432 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143303    2546 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143306    2546 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143309    2546 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143312    2546 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143314    2546 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143317    2546 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143319    2546 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143322    2546 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143325    2546 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143327    2546 feature_gate.go:328] unrecognized feature gate: Example2
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143330    2546 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143332    2546 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143335    2546 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143338    2546 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143341    2546 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143343    2546 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143346    2546 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143348    2546 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143351    2546 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143353    2546 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 17 21:17:03.149895 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143356    2546 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:03.143358    2546 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.143364    2546 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.143455    2546 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.145498    2546 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.147501    2546 server.go:1019] "Starting client certificate rotation"
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.147596    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.147629    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 17 21:17:03.150488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.150107    2546 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:17:03.151174 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.151156    2546 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 17 21:17:03.161189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.161169    2546 log.go:25] "Validated CRI v1 runtime API"
Jul 17 21:17:03.170569 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.170551    2546 log.go:25] "Validated CRI v1 image API"
Jul 17 21:17:03.172116 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.172102    2546 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 17 21:17:03.176660 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.176641    2546 fs.go:135] Filesystem UUIDs: map[2aa2d6c4-b05f-4d97-abfa-213f18dd771c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 a558c4f6-d816-47bf-a474-a2ed1e6ccfdc:/dev/nvme0n1p4]
Jul 17 21:17:03.176733 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.176659    2546 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 17 21:17:03.181649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.181543    2546 manager.go:216] Machine: {Timestamp:2026-07-17 21:17:03.180175809 +0000 UTC m=+0.339975469 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3090580 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec253fb9be94877fe1e436814cad937c SystemUUID:ec253fb9-be94-877f-e1e4-36814cad937c BootID:80029ed5-1d9d-468e-8939-34da4b0c8484 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:46:f0:d7:18:d9 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:46:f0:d7:18:d9 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b6:db:f9:4b:0c:ab Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 17 21:17:03.181649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.181645    2546 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 17 21:17:03.181749 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.181710    2546 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 17 21:17:03.184908 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.184880    2546 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 17 21:17:03.185039 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.184911    2546 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-135-220.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 17 21:17:03.185631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.185621    2546 topology_manager.go:138] "Creating topology manager with none policy"
Jul 17 21:17:03.185667 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.185634    2546 container_manager_linux.go:309] "Creating device plugin manager"
Jul 17 21:17:03.185667 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.185641    2546 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:17:03.186508 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.186496    2546 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 17 21:17:03.187822 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.187812    2546 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 17 21:17:03.189584 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.189569    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:17:03.191252 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191241    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:17:03.191372 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191362    2546 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 17 21:17:03.191407 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191393    2546 kubelet.go:486] "Attempting to sync node with API server"
Jul 17 21:17:03.191407 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191403    2546 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 17 21:17:03.191459 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191417    2546 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 17 21:17:03.191459 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191426    2546 kubelet.go:398] "Adding apiserver pod source"
Jul 17 21:17:03.191459 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191435    2546 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 17 21:17:03.191762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191751    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:17:03.191794 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.191769    2546 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 17 21:17:03.193973 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.193957    2546 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 17 21:17:03.196007 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.195994    2546 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 17 21:17:03.196053 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.196025    2546 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 17 21:17:03.198118 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198106    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 17 21:17:03.198151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198124    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 17 21:17:03.198151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198130    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 17 21:17:03.198151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198136    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 17 21:17:03.198151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198144    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 17 21:17:03.198151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198149    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198157    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198162    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198168    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198173    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198189    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 17 21:17:03.198314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198194    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 17 21:17:03.198826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198814    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 17 21:17:03.198858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.198829    2546 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 17 21:17:03.201056 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.201005    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-135-220.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 17 21:17:03.201056 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.201007    2546 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 17 21:17:03.201143 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.201090    2546 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-135-220.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 17 21:17:03.202517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.202506    2546 server.go:1268] "Started kubelet"
Jul 17 21:17:03.202621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.202602    2546 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 17 21:17:03.202790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.202735    2546 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 17 21:17:03.202883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.202809    2546 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 17 21:17:03.208072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.208051    2546 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 17 21:17:03.208502 ip-10-0-135-220 systemd[1]: Started Kubernetes Kubelet.
Jul 17 21:17:03.209267 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.208953    2546 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 17 21:17:03.210219 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.210202    2546 server.go:310] "Adding debug handlers to kubelet server"
Jul 17 21:17:03.212070 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.212044    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w8tsr"
Jul 17 21:17:03.213046 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.212040    2546 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-135-220.ec2.internal.18c33055fa88e952  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-135-220.ec2.internal,UID:ip-10-0-135-220.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-135-220.ec2.internal,},FirstTimestamp:2026-07-17 21:17:03.202478418 +0000 UTC m=+0.362278078,LastTimestamp:2026-07-17 21:17:03.202478418 +0000 UTC m=+0.362278078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-135-220.ec2.internal,}"
Jul 17 21:17:03.217206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217187    2546 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 17 21:17:03.217206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217204    2546 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 17 21:17:03.217369 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.217351    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.217858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217777    2546 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 17 21:17:03.217858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217795    2546 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 17 21:17:03.217995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217866    2546 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 17 21:17:03.217995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217920    2546 reconstruct.go:97] "Volume reconstruction finished"
Jul 17 21:17:03.217995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.217928    2546 reconciler.go:29] "Reconciler: start to sync state"
Jul 17 21:17:03.218101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218035    2546 factory.go:55] Registering systemd factory
Jul 17 21:17:03.218101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218063    2546 factory.go:223] Registration of the systemd container factory successfully
Jul 17 21:17:03.218282 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218267    2546 factory.go:153] Registering CRI-O factory
Jul 17 21:17:03.218374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218284    2546 factory.go:223] Registration of the crio container factory successfully
Jul 17 21:17:03.218374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218353    2546 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 17 21:17:03.218464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218375    2546 factory.go:103] Registering Raw factory
Jul 17 21:17:03.218464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218390    2546 manager.go:1196] Started watching for new ooms in manager
Jul 17 21:17:03.218960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.218945    2546 manager.go:349] Starting recovery of all containers
Jul 17 21:17:03.220193 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.220175    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-w8tsr"
Jul 17 21:17:03.229436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.229273    2546 manager.go:354] Recovery completed
Jul 17 21:17:03.229523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.229367    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:03.233479 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.233467    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.236012 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.235997    2546 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-135-220.ec2.internal\" not found" node="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.236555 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.236541    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.236596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.236575    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.236596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.236585    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.237056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237040    2546 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 17 21:17:03.237056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237050    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.237056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237055    2546 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 17 21:17:03.237197 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237071    2546 state_mem.go:36] "Initialized new in-memory state store"
Jul 17 21:17:03.237772 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237756    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.237857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237788    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.237857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.237804    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.238345 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.238332    2546 policy_none.go:49] "None policy: Start"
Jul 17 21:17:03.238390 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.238348    2546 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 17 21:17:03.238390 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.238358    2546 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 17 21:17:03.239568 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.239553    2546 policy_none.go:47] "Start"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.271992    2546 manager.go:337] "Starting Device Plugin manager"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.272015    2546 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272024    2546 server.go:87] "Starting device plugin registration server"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272210    2546 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272231    2546 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272329    2546 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272422    2546 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.272432    2546 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.272830    2546 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 17 21:17:03.289417 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.272882    2546 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.349499 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.349457    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 17 21:17:03.350836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.350818    2546 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 17 21:17:03.350886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.350868    2546 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 17 21:17:03.350918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.350896    2546 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 17 21:17:03.350961 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.350937    2546 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 17 21:17:03.353374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.353353    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:03.372508 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.372492    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.373374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.373320    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.373374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.373353    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.373374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.373365    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.373499 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.373396    2546 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.381490 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.381474    2546 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.381589 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.381499    2546 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-135-220.ec2.internal\": node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.397854 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.397838    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.451228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.451205    2546 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"]
Jul 17 21:17:03.451331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.451274    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.452839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.452826    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.452893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.452852    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.452893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.452862    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.454131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.454117    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.454256 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.454241    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.454328 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.454275    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.455034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455012    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.455115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455018    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.455115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455048    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.455115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455061    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.455115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455064    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.455115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.455076    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.456471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.456453    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.456537 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.456493    2546 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 17 21:17:03.457180 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.457167    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientMemory"
Jul 17 21:17:03.457261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.457194    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasNoDiskPressure"
Jul 17 21:17:03.457261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.457205    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeHasSufficientPID"
Jul 17 21:17:03.485767 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.485744    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-220.ec2.internal\" not found" node="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.489985 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.489967    2546 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-135-220.ec2.internal\" not found" node="ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.497946 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.497932    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.520018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.519997    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.520076 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.520024    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.520076 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.520040    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3d415bcf690ad6d0665b9c29c5a5dbf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-220.ec2.internal\" (UID: \"d3d415bcf690ad6d0665b9c29c5a5dbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.598303 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.598271    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.620633 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620611    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.620698 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620636    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.620698 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620652    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3d415bcf690ad6d0665b9c29c5a5dbf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-220.ec2.internal\" (UID: \"d3d415bcf690ad6d0665b9c29c5a5dbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.620762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620698    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/d3d415bcf690ad6d0665b9c29c5a5dbf-config\") pod \"kube-apiserver-proxy-ip-10-0-135-220.ec2.internal\" (UID: \"d3d415bcf690ad6d0665b9c29c5a5dbf\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.620762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620711    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.620762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.620735    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7521c5b90c8a59dd7dbee27a30e1615f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal\" (UID: \"7521c5b90c8a59dd7dbee27a30e1615f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.699040 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.698993    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.789477 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.789444    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.793349 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:03.793334    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:03.799266 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.799247    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:03.899771 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:03.899734    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.000315 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.000244    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.100712 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.100682    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.147213 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.147181    2546 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 17 21:17:04.147783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.147342    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:17:04.147783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.147351    2546 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 17 21:17:04.201709 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.201675    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.225815 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.225762    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-16 21:12:04 +0000 UTC" deadline="2028-05-03 02:36:55.17766233 +0000 UTC"
Jul 17 21:17:04.225815 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.225809    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15725h19m50.951856127s"
Jul 17 21:17:04.302128 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.302062    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.333578 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:04.333541    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d415bcf690ad6d0665b9c29c5a5dbf.slice/crio-5b45c8f124d76d97a4b10f061d20921adaffd3b9c6213aaecf58e129e32ad79c WatchSource:0}: Error finding container 5b45c8f124d76d97a4b10f061d20921adaffd3b9c6213aaecf58e129e32ad79c: Status 404 returned error can't find the container with id 5b45c8f124d76d97a4b10f061d20921adaffd3b9c6213aaecf58e129e32ad79c
Jul 17 21:17:04.337586 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.337570    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:17:04.354643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:04.354593    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal" event={"ID":"d3d415bcf690ad6d0665b9c29c5a5dbf","Type":"ContainerStarted","Data":"5b45c8f124d76d97a4b10f061d20921adaffd3b9c6213aaecf58e129e32ad79c"}
Jul 17 21:17:04.402833 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.402806    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:04.440700 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:04.440678    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7521c5b90c8a59dd7dbee27a30e1615f.slice/crio-c26eccd1e2e1fd4ce99ba78d077c4fbc9974b3a00da0322ef91552cb35c0a780 WatchSource:0}: Error finding container c26eccd1e2e1fd4ce99ba78d077c4fbc9974b3a00da0322ef91552cb35c0a780: Status 404 returned error can't find the container with id c26eccd1e2e1fd4ce99ba78d077c4fbc9974b3a00da0322ef91552cb35c0a780
Jul 17 21:17:04.503609 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:04.503583    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:05.783553 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:05.783533    2546 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-135-220.ec2.internal\" not found"
Jul 17 21:17:05.800905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.800884    2546 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:05.897656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.897635    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:05.908647 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.908627    2546 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:05.912977 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.912963    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:17:05.913936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.913923    2546 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal"
Jul 17 21:17:05.925379 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:05.925367    2546 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 17 21:17:06.361849 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.361694    2546 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:06.372234 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.372210    2546 apiserver.go:52] "Watching apiserver"
Jul 17 21:17:06.381744 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.381407    2546 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 17 21:17:06.383128 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.383099    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-dktcq","openshift-network-diagnostics/network-check-target-qrm7f","openshift-network-operator/iptables-alerter-tq8x7","openshift-ovn-kubernetes/ovnkube-node-49bsn","kube-system/konnectivity-agent-jsttb","kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4","openshift-image-registry/node-ca-vqd2w","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal","openshift-multus/multus-9jpfn","openshift-multus/multus-additional-cni-plugins-9cgfc","openshift-multus/network-metrics-daemon-p8c6r","kube-system/global-pull-secret-syncer-mwjff","openshift-cluster-node-tuning-operator/tuned-6kztn"]
Jul 17 21:17:06.384813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.384794    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.386694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.386435    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.386694 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.386515    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:06.388112 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.387901    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.389912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.389712    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 17 21:17:06.390135 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.390113    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.393676 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.392543    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.393676 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.392543    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.395486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.394407    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 17 21:17:06.395486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.394471    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-z7bfd\""
Jul 17 21:17:06.395486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.394685    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.395486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.395442    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.395760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.395732    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.396898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.396240    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.396898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.396414    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 17 21:17:06.396898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.396531    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tk7km\""
Jul 17 21:17:06.396898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.396589    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 17 21:17:06.396898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.396791    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 17 21:17:06.397206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.397108    2546 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 17 21:17:06.397866 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.397432    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.398481 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.398440    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-zxg6r\""
Jul 17 21:17:06.399282 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.398860    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:06.399282 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.398924    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:06.400423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.399873    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.400423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.399928    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.400423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.400081    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.400423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.400133    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 17 21:17:06.400635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.400610    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.400755    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.400879    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401072    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401082    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401246    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401272    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-pdsk9\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401429    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-574l5\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401510    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-xntmp\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401642    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-56bcr\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401704    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401844    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 17 21:17:06.402520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.401928    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.403229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.402654    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.403229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.403146    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.404434 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.404420    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.404519 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.404481    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:06.406764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.406743    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-f9rlz\""
Jul 17 21:17:06.406975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.406956    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 17 21:17:06.407080 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.407034    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 17 21:17:06.407639 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.407620    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 17 21:17:06.407639 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.407628    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 17 21:17:06.408690 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408671    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-etc-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.408797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-run\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.408797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408776    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-host\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.408900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408800    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-tuned\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.408900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408822    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-kubelet\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.408900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408843    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-node-log\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.408900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-env-overrides\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.408900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408888    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p9ls\" (UniqueName: \"kubernetes.io/projected/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kube-api-access-4p9ls\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408922    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-config\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408943    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-modprobe-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408963    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-ovn\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.408981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409015    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-hosts-file\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/730765d7-ae92-4d45-b064-3ab24aeac23d-kube-api-access-pmrpj\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409070    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409085    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-var-lib-kubelet\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.409120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-registration-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409139    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/730765d7-ae92-4d45-b064-3ab24aeac23d-iptables-alerter-script\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409161    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-dbus\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409175    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc74f1-96b7-474e-9fe5-cb1128657650-serviceca\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-slash\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409212    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-log-socket\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409233    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-script-lib\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409259    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysconfig\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409325    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtprc\" (UniqueName: \"kubernetes.io/projected/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-kube-api-access-wtprc\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.409392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409372    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409401    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-lib-modules\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409431    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-systemd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409475    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-kubelet-config\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409535    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409569    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-bin\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409594    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovn-node-metrics-cert\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409617    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6cgt\" (UniqueName: \"kubernetes.io/projected/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-kube-api-access-z6cgt\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409640    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409662    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409685    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-netd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-tmp-dir\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.409759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409756    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-socket-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409781    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc74f1-96b7-474e-9fe5-cb1128657650-host\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409811    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-netns\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409833    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-systemd\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409856    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-kubernetes\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409877    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-tmp\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0620747-a811-4560-942d-1cb176ff1af5-agent-certs\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409929    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.409972    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-sys\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410022    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-device-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410055    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410085    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdx4t\" (UniqueName: \"kubernetes.io/projected/54dc74f1-96b7-474e-9fe5-cb1128657650-kube-api-access-sdx4t\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410099    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/730765d7-ae92-4d45-b064-3ab24aeac23d-host-slash\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410112    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-systemd-units\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410126    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wrg2\" (UniqueName: \"kubernetes.io/projected/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-kube-api-access-7wrg2\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410152    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-var-lib-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.410292 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410192    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0620747-a811-4560-942d-1cb176ff1af5-konnectivity-ca\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.410884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410218    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-conf\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.410884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.410254    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-sys-fs\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.411856 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.411797    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 17 21:17:06.416195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.414933    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-kgl8q\""
Jul 17 21:17:06.416307 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.416239    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 17 21:17:06.418195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.418173    2546 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 17 21:17:06.445495 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.445461    2546 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8sqt"
Jul 17 21:17:06.453998 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.453961    2546 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-v8sqt"
Jul 17 21:17:06.498261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.498230    2546 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 17 21:17:06.504125 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.504096    2546 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 17 21:17:06.510477 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p9ls\" (UniqueName: \"kubernetes.io/projected/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kube-api-access-4p9ls\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.510658 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510486    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-config\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.510658 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510513    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-modprobe-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.510658 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510647    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-modprobe-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.510820 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510760    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.510820 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510791    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-ovn\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.510920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510819    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.510920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510843    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-hosts-file\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.510920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510868    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qw97h\" (UniqueName: \"kubernetes.io/projected/fc22caa7-9281-44f6-9738-4edffb7a5237-kube-api-access-qw97h\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.510920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510871    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-ovn\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.510920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510893    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/730765d7-ae92-4d45-b064-3ab24aeac23d-kube-api-access-pmrpj\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510933    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kubelet-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510933    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-hosts-file\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510979    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-var-lib-kubelet\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511008    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-var-lib-kubelet\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511006    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-kubelet\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.510967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511028    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-cnibin\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511051    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511068    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-registration-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511083    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/730765d7-ae92-4d45-b064-3ab24aeac23d-iptables-alerter-script\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511111    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-socket-dir-parent\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-etc-kubernetes\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511147    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-dbus\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.511153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511157    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-registration-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc74f1-96b7-474e-9fe5-cb1128657650-serviceca\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511204    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-slash\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511251    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-config\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511303    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-slash\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511311    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-log-socket\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511344    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-script-lib\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511355    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-log-socket\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511374    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysconfig\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511394    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtprc\" (UniqueName: \"kubernetes.io/projected/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-kube-api-access-wtprc\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511419    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511444    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-lib-modules\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511462    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysconfig\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511469    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-os-release\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511508    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-cni-binary-copy\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511534    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-netns\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511557    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-hostroot\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511581    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-daemon-config\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.511872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511608    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-systemd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511353    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-dbus\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511634    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/730765d7-ae92-4d45-b064-3ab24aeac23d-iptables-alerter-script\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511608    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/54dc74f1-96b7-474e-9fe5-cb1128657650-serviceca\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511634    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-os-release\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511680    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-systemd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511696    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-kubelet-config\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511738    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511755    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/aff86450-cf63-48d6-8ceb-d720e646190d-kubelet-config\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511779    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-bin\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511796    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-lib-modules\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovn-node-metrics-cert\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511815    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-ovn-kubernetes\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511828    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6cgt\" (UniqueName: \"kubernetes.io/projected/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-kube-api-access-z6cgt\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-bin\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511872    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhkhs\" (UniqueName: \"kubernetes.io/projected/b196a3d2-d251-432c-bcfd-abb6171d087b-kube-api-access-lhkhs\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511896    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovnkube-script-lib\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.512674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511900    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn4hf\" (UniqueName: \"kubernetes.io/projected/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-kube-api-access-vn4hf\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511950    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.511980    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-netd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-run-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512031    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-tmp-dir\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512066    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-cni-netd\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.512077    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512095    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-system-cni-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.512154    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:07.012131454 +0000 UTC m=+2.992501927 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512144    2546 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512178    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-socket-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512207    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc74f1-96b7-474e-9fe5-cb1128657650-host\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512234    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-netns\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512258    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-systemd\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512285    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-conf-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512311    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-multus-certs\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.513454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512286    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/54dc74f1-96b7-474e-9fe5-cb1128657650-host\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512313    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-socket-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512319    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-run-netns\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-tmp-dir\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512355    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-systemd\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512358    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-kubernetes\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512440    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-tmp\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512446    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-kubernetes\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512470    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-system-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512498    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-k8s-cni-cncf-io\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512529    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512577    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512602    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0620747-a811-4560-942d-1cb176ff1af5-agent-certs\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512641    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512672    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-sys\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512697    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-device-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512737    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.514446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512764    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdx4t\" (UniqueName: \"kubernetes.io/projected/54dc74f1-96b7-474e-9fe5-cb1128657650-kube-api-access-sdx4t\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512786    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/730765d7-ae92-4d45-b064-3ab24aeac23d-host-slash\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512807    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-systemd-units\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wrg2\" (UniqueName: \"kubernetes.io/projected/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-kube-api-access-7wrg2\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512870    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-multus\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512894    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512920    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-var-lib-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512952    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0620747-a811-4560-942d-1cb176ff1af5-konnectivity-ca\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512974    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-conf\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.512994    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-bin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513014    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-sys-fs\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513034    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-etc-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513057    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-run\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-host\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513096    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-tuned\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513118    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513136    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-d\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.515285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-kubelet\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513163    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-node-log\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513192    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-sys\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513200    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-env-overrides\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513220    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-cnibin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513236    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-device-dir\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513034    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-systemd-units\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513328    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-sys-fs\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513341    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8a8cf97a-2109-4a63-84b5-69bdf583dd05-etc-selinux\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-run\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513365    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-etc-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513397    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-var-lib-openvswitch\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513526    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-host\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513599    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/730765d7-ae92-4d45-b064-3ab24aeac23d-host-slash\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513638    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-host-kubelet\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513642    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-node-log\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.513869    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/c0620747-a811-4560-942d-1cb176ff1af5-konnectivity-ca\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.514083    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-env-overrides\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.516117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.514227    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-sysctl-conf\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.517184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.516062    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-etc-tuned\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.517184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.516476    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-tmp\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.517184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.517051    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-ovn-node-metrics-cert\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.518891 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.518288    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/c0620747-a811-4560-942d-1cb176ff1af5-agent-certs\") pod \"konnectivity-agent-jsttb\" (UID: \"c0620747-a811-4560-942d-1cb176ff1af5\") " pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.520657 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.520637    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p9ls\" (UniqueName: \"kubernetes.io/projected/8a8cf97a-2109-4a63-84b5-69bdf583dd05-kube-api-access-4p9ls\") pod \"aws-ebs-csi-driver-node-ggqj4\" (UID: \"8a8cf97a-2109-4a63-84b5-69bdf583dd05\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.520790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.520780    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmrpj\" (UniqueName: \"kubernetes.io/projected/730765d7-ae92-4d45-b064-3ab24aeac23d-kube-api-access-pmrpj\") pod \"iptables-alerter-tq8x7\" (UID: \"730765d7-ae92-4d45-b064-3ab24aeac23d\") " pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.521378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.521357    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6cgt\" (UniqueName: \"kubernetes.io/projected/e1dda6d1-c0dd-4780-86bb-bd3f309293e6-kube-api-access-z6cgt\") pod \"tuned-6kztn\" (UID: \"e1dda6d1-c0dd-4780-86bb-bd3f309293e6\") " pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.522209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.522187    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtprc\" (UniqueName: \"kubernetes.io/projected/51dc316d-79fd-4a8d-9f06-e44e9e84e2e5-kube-api-access-wtprc\") pod \"node-resolver-dktcq\" (UID: \"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5\") " pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.522746 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.522702    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:06.522746 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.522739    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:06.522746 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.522751    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:06.522925 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.522808    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:07.022789313 +0000 UTC m=+3.003159786 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:06.525068 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.525034    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdx4t\" (UniqueName: \"kubernetes.io/projected/54dc74f1-96b7-474e-9fe5-cb1128657650-kube-api-access-sdx4t\") pod \"node-ca-vqd2w\" (UID: \"54dc74f1-96b7-474e-9fe5-cb1128657650\") " pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.525175 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.525129    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wrg2\" (UniqueName: \"kubernetes.io/projected/d347b8ae-30f7-41b6-b5a9-b3adbb9f2299-kube-api-access-7wrg2\") pod \"ovnkube-node-49bsn\" (UID: \"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299\") " pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.537663 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.537615    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal" event={"ID":"7521c5b90c8a59dd7dbee27a30e1615f","Type":"ContainerStarted","Data":"c26eccd1e2e1fd4ce99ba78d077c4fbc9974b3a00da0322ef91552cb35c0a780"}
Jul 17 21:17:06.614010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.613926    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-multus\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.613966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.614010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.613994    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-bin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614014    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614034    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-cnibin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614052    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614071    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qw97h\" (UniqueName: \"kubernetes.io/projected/fc22caa7-9281-44f6-9738-4edffb7a5237-kube-api-access-qw97h\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614094    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-kubelet\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614108    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-cnibin\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614122    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614138    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-socket-dir-parent\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614153    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-etc-kubernetes\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614186    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-os-release\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614201    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-cni-binary-copy\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614217    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-netns\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614231    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-hostroot\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614252    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-daemon-config\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614268    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-os-release\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.614285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614288    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhkhs\" (UniqueName: \"kubernetes.io/projected/b196a3d2-d251-432c-bcfd-abb6171d087b-kube-api-access-lhkhs\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn4hf\" (UniqueName: \"kubernetes.io/projected/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-kube-api-access-vn4hf\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614355    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-system-cni-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614382    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-conf-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614406    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-multus-certs\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614429    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-system-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-k8s-cni-cncf-io\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614476    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614497    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.614618    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:06.614664    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:07.114648507 +0000 UTC m=+3.095018962 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:06.615004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.614867    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-multus\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.615445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-binary-copy\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.615445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-cni-bin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615602    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-conf-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615613    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-hostroot\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615632    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-k8s-cni-cncf-io\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615671    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-cnibin\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615702    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-multus-certs\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615751    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615780    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-system-cni-dir\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615874    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.615956    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-os-release\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616017    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-socket-dir-parent\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616024    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-etc-kubernetes\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616070    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-cnibin\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616093    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-os-release\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616101    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-var-lib-kubelet\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616140    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-system-cni-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616169    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-multus-daemon-config\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616180    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fc22caa7-9281-44f6-9738-4edffb7a5237-host-run-netns\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.616471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616236    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b196a3d2-d251-432c-bcfd-abb6171d087b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.617290 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616443    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fc22caa7-9281-44f6-9738-4edffb7a5237-cni-binary-copy\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.617290 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.616743    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/b196a3d2-d251-432c-bcfd-abb6171d087b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.627299 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.627264    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhkhs\" (UniqueName: \"kubernetes.io/projected/b196a3d2-d251-432c-bcfd-abb6171d087b-kube-api-access-lhkhs\") pod \"multus-additional-cni-plugins-9cgfc\" (UID: \"b196a3d2-d251-432c-bcfd-abb6171d087b\") " pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:06.627964 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.627944    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn4hf\" (UniqueName: \"kubernetes.io/projected/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-kube-api-access-vn4hf\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:06.628060 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.628007    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qw97h\" (UniqueName: \"kubernetes.io/projected/fc22caa7-9281-44f6-9738-4edffb7a5237-kube-api-access-qw97h\") pod \"multus-9jpfn\" (UID: \"fc22caa7-9281-44f6-9738-4edffb7a5237\") " pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.697736 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.697674    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4"
Jul 17 21:17:06.707582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.707551    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-tq8x7"
Jul 17 21:17:06.717700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.717670    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:06.723684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.723656    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:06.730702 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.730680    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-6kztn"
Jul 17 21:17:06.738567 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.738547    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-vqd2w"
Jul 17 21:17:06.747767 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.747743    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-dktcq"
Jul 17 21:17:06.756588 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.756558    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9jpfn"
Jul 17 21:17:06.762668 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:06.762650    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-9cgfc"
Jul 17 21:17:07.018106 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.018024    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:07.018260 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.018170    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:07.018260 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.018229    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:08.018212077 +0000 UTC m=+3.998582532 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:07.119287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.119255    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:07.119440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.119314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:07.119440 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119403    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:07.119440 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119407    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:07.119440 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119428    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:07.119440 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119441    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:07.119596 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119470    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:08.119452924 +0000 UTC m=+4.099823379 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:07.119596 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:07.119487    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:08.119477495 +0000 UTC m=+4.099847951 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:07.419080 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.419035    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a8cf97a_2109_4a63_84b5_69bdf583dd05.slice/crio-9df7ce80cc20e34e08ffda7d4d7ad1da0eb7792582d18874e9e097b147bded5d WatchSource:0}: Error finding container 9df7ce80cc20e34e08ffda7d4d7ad1da0eb7792582d18874e9e097b147bded5d: Status 404 returned error can't find the container with id 9df7ce80cc20e34e08ffda7d4d7ad1da0eb7792582d18874e9e097b147bded5d
Jul 17 21:17:07.422405 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.422378    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51dc316d_79fd_4a8d_9f06_e44e9e84e2e5.slice/crio-94060639c7bf8666eed782155158b433737990efd2d8689bb9424e87a49cfed7 WatchSource:0}: Error finding container 94060639c7bf8666eed782155158b433737990efd2d8689bb9424e87a49cfed7: Status 404 returned error can't find the container with id 94060639c7bf8666eed782155158b433737990efd2d8689bb9424e87a49cfed7
Jul 17 21:17:07.426137 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.424757    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod730765d7_ae92_4d45_b064_3ab24aeac23d.slice/crio-28b6a280f19c1f14678b3f9dfacb9b315455fa9ebbe21a6f31e0d65cda8f4bb3 WatchSource:0}: Error finding container 28b6a280f19c1f14678b3f9dfacb9b315455fa9ebbe21a6f31e0d65cda8f4bb3: Status 404 returned error can't find the container with id 28b6a280f19c1f14678b3f9dfacb9b315455fa9ebbe21a6f31e0d65cda8f4bb3
Jul 17 21:17:07.426137 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.425452    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb196a3d2_d251_432c_bcfd_abb6171d087b.slice/crio-086c2097f898a6ed1d6bbe24940e2a23c7b30d83da0faf3b0e7d3817a7ee2c9e WatchSource:0}: Error finding container 086c2097f898a6ed1d6bbe24940e2a23c7b30d83da0faf3b0e7d3817a7ee2c9e: Status 404 returned error can't find the container with id 086c2097f898a6ed1d6bbe24940e2a23c7b30d83da0faf3b0e7d3817a7ee2c9e
Jul 17 21:17:07.426275 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.426257    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1dda6d1_c0dd_4780_86bb_bd3f309293e6.slice/crio-deadfe81216c91d9ccf8f56f5ee7846407d5a358527abf003c63b26100dc5af2 WatchSource:0}: Error finding container deadfe81216c91d9ccf8f56f5ee7846407d5a358527abf003c63b26100dc5af2: Status 404 returned error can't find the container with id deadfe81216c91d9ccf8f56f5ee7846407d5a358527abf003c63b26100dc5af2
Jul 17 21:17:07.427291 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.427267    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54dc74f1_96b7_474e_9fe5_cb1128657650.slice/crio-fdf4b3e2242efeed233b7980000c291460a5679f4989f216c157c16ae2d96651 WatchSource:0}: Error finding container fdf4b3e2242efeed233b7980000c291460a5679f4989f216c157c16ae2d96651: Status 404 returned error can't find the container with id fdf4b3e2242efeed233b7980000c291460a5679f4989f216c157c16ae2d96651
Jul 17 21:17:07.428448 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.428116    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0620747_a811_4560_942d_1cb176ff1af5.slice/crio-2e1bb6abbfccad599445d49d62822c0a81cb21e7bd6d5dc551fbbeb11b4f6c5d WatchSource:0}: Error finding container 2e1bb6abbfccad599445d49d62822c0a81cb21e7bd6d5dc551fbbeb11b4f6c5d: Status 404 returned error can't find the container with id 2e1bb6abbfccad599445d49d62822c0a81cb21e7bd6d5dc551fbbeb11b4f6c5d
Jul 17 21:17:07.431180 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.429310    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc22caa7_9281_44f6_9738_4edffb7a5237.slice/crio-1581fcea5e0103b4a45aac740e0cdb02635380808b04c62f91bdc1b3a7554287 WatchSource:0}: Error finding container 1581fcea5e0103b4a45aac740e0cdb02635380808b04c62f91bdc1b3a7554287: Status 404 returned error can't find the container with id 1581fcea5e0103b4a45aac740e0cdb02635380808b04c62f91bdc1b3a7554287
Jul 17 21:17:07.431438 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:07.431418    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd347b8ae_30f7_41b6_b5a9_b3adbb9f2299.slice/crio-5cdd0837160c3b99d665ff5bd1aa4fdb50ae90b115e5ab193c2febcee6c03122 WatchSource:0}: Error finding container 5cdd0837160c3b99d665ff5bd1aa4fdb50ae90b115e5ab193c2febcee6c03122: Status 404 returned error can't find the container with id 5cdd0837160c3b99d665ff5bd1aa4fdb50ae90b115e5ab193c2febcee6c03122
Jul 17 21:17:07.457572 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.457398    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:12:06 +0000 UTC" deadline="2028-01-08 02:48:14.318212965 +0000 UTC"
Jul 17 21:17:07.457572 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.457539    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12941h31m6.860681332s"
Jul 17 21:17:07.540746 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.540704    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dktcq" event={"ID":"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5","Type":"ContainerStarted","Data":"94060639c7bf8666eed782155158b433737990efd2d8689bb9424e87a49cfed7"}
Jul 17 21:17:07.542359 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.542330    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4" event={"ID":"8a8cf97a-2109-4a63-84b5-69bdf583dd05","Type":"ContainerStarted","Data":"9df7ce80cc20e34e08ffda7d4d7ad1da0eb7792582d18874e9e097b147bded5d"}
Jul 17 21:17:07.544249 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.544225    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal" event={"ID":"d3d415bcf690ad6d0665b9c29c5a5dbf","Type":"ContainerStarted","Data":"594399f61b0e8b8e743c161925a80474db202dd6c48a564aa170e4c757c5ca0b"}
Jul 17 21:17:07.546179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.546153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerStarted","Data":"086c2097f898a6ed1d6bbe24940e2a23c7b30d83da0faf3b0e7d3817a7ee2c9e"}
Jul 17 21:17:07.547226 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.547206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"5cdd0837160c3b99d665ff5bd1aa4fdb50ae90b115e5ab193c2febcee6c03122"}
Jul 17 21:17:07.549077 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.549055    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9jpfn" event={"ID":"fc22caa7-9281-44f6-9738-4edffb7a5237","Type":"ContainerStarted","Data":"1581fcea5e0103b4a45aac740e0cdb02635380808b04c62f91bdc1b3a7554287"}
Jul 17 21:17:07.551991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.551966    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqd2w" event={"ID":"54dc74f1-96b7-474e-9fe5-cb1128657650","Type":"ContainerStarted","Data":"fdf4b3e2242efeed233b7980000c291460a5679f4989f216c157c16ae2d96651"}
Jul 17 21:17:07.553673 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.553650    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kztn" event={"ID":"e1dda6d1-c0dd-4780-86bb-bd3f309293e6","Type":"ContainerStarted","Data":"deadfe81216c91d9ccf8f56f5ee7846407d5a358527abf003c63b26100dc5af2"}
Jul 17 21:17:07.555065 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.555048    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jsttb" event={"ID":"c0620747-a811-4560-942d-1cb176ff1af5","Type":"ContainerStarted","Data":"2e1bb6abbfccad599445d49d62822c0a81cb21e7bd6d5dc551fbbeb11b4f6c5d"}
Jul 17 21:17:07.556460 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.556442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq8x7" event={"ID":"730765d7-ae92-4d45-b064-3ab24aeac23d","Type":"ContainerStarted","Data":"28b6a280f19c1f14678b3f9dfacb9b315455fa9ebbe21a6f31e0d65cda8f4bb3"}
Jul 17 21:17:07.560157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:07.560095    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-135-220.ec2.internal" podStartSLOduration=2.560075302 podStartE2EDuration="2.560075302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:07.559610135 +0000 UTC m=+3.539980611" watchObservedRunningTime="2026-07-17 21:17:07.560075302 +0000 UTC m=+3.540445761"
Jul 17 21:17:08.025567 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.025529    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:08.025769 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.025681    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:08.025769 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.025758    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:10.025739658 +0000 UTC m=+6.006110127 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:08.126480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.126445    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:08.126663 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.126517    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:08.126663 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126633    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:08.126821 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126693    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:10.126674237 +0000 UTC m=+6.107044698 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:08.126821 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126802    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:08.126821 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126817    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:08.126985 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126829    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:08.126985 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.126869    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:10.126856502 +0000 UTC m=+6.107226964 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:08.458619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.458510    2546 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-16 21:12:06 +0000 UTC" deadline="2028-04-19 18:38:44.183063596 +0000 UTC"
Jul 17 21:17:08.458619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.458567    2546 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15405h21m35.724499661s"
Jul 17 21:17:08.533909 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.533880    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:08.534076 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.534003    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:08.534401 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.534384    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:08.534514 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.534485    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:08.534571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.534554    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:08.534643 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:08.534626    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:08.571802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.571767    2546 generic.go:356] "Generic (PLEG): container finished" podID="7521c5b90c8a59dd7dbee27a30e1615f" containerID="50b3f466bb688d3403df525844ba97990178011e1987f7e43ecf412f6ca1ab2d" exitCode=0
Jul 17 21:17:08.572277 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:08.572256    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal" event={"ID":"7521c5b90c8a59dd7dbee27a30e1615f","Type":"ContainerDied","Data":"50b3f466bb688d3403df525844ba97990178011e1987f7e43ecf412f6ca1ab2d"}
Jul 17 21:17:09.586356 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:09.585688    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal" event={"ID":"7521c5b90c8a59dd7dbee27a30e1615f","Type":"ContainerStarted","Data":"e71a44d51f53367014d4bc4fee412bb5b4a90c248e62cf51fb5d06cb4f06e937"}
Jul 17 21:17:09.603733 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:09.603435    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-135-220.ec2.internal" podStartSLOduration=4.603412743 podStartE2EDuration="4.603412743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:09.60299612 +0000 UTC m=+5.583366598" watchObservedRunningTime="2026-07-17 21:17:09.603412743 +0000 UTC m=+5.583783222"
Jul 17 21:17:10.046629 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.046545    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:10.046807 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.046712    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:10.046807 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.046791    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:14.046770529 +0000 UTC m=+10.027140987 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.147144    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.147216    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147368    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147401    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147428    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147445    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147451    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:14.147429834 +0000 UTC m=+10.127800305 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:10.147808 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.147512    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:14.14749423 +0000 UTC m=+10.127864688 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.533511    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.533651    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.533713    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.533813    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:10.533863    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:10.534319 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:10.533928    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:12.533165 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:12.533132    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:12.533597 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:12.533261    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:12.533707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:12.533666    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:12.533806 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:12.533779    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:12.533865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:12.533846    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:12.533963 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:12.533931    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:14.075983 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.075470    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:14.075983 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.075605    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:14.075983 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.075677    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:22.07565985 +0000 UTC m=+18.056030326 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:14.176302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.176268    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:14.176434 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.176321    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:14.176482 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176436    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:14.176482 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176451    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:14.176482 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176460    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:14.176568 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176508    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:22.176495897 +0000 UTC m=+18.156866351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:14.176927 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176840    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:14.176927 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.176896    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:22.176880719 +0000 UTC m=+18.157251177 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:14.531887 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.531810    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:14.532039 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.531920    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:14.532297 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.532275    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:14.532410 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.532389    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:14.532636 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:14.532523    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:14.532636 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:14.532605    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:16.530818    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:16.530952    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:16.530817    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:16.531084    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:16.531128    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:16.531297 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:16.531194    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:18.531186 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:18.531150    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:18.531602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:18.531150    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:18.531602 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:18.531290    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:18.531602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:18.531350    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:18.531602 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:18.531472    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:18.531602 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:18.531536    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:20.531250 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:20.531219    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:20.531250 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:20.531234    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:20.531756 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:20.531219    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:20.531756 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:20.531348    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:20.531756 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:20.531417    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:20.531756 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:20.531493    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:22.137442 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.137402    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:22.137847 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.137558    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:22.137847 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.137622    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:17:38.137606717 +0000 UTC m=+34.117977171 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:22.238502 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.238463    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:22.238651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.238525    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:22.238651 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238632    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:22.238651 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238639    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:22.238841 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238655    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:22.238841 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238668    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:22.238841 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238694    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:17:38.238680284 +0000 UTC m=+34.219050738 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:22.238841 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.238736    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:17:38.238701526 +0000 UTC m=+34.219071996 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.531055    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.531068    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.531184    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.531257    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:22.531324    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:22.531554 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:22.531397    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:24.532013 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:24.531981    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:24.532434 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:24.532078    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:24.532434 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:24.532113    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:24.532434 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:24.532157    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:24.532434 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:24.532198    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:24.532434 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:24.532274    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:26.531836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.531596    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:26.532644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.531623    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:26.532644 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:26.531924    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:26.532644 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:26.532051    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:26.532644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.531678    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:26.532644 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:26.532199    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:26.625301 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.625165    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9jpfn" event={"ID":"fc22caa7-9281-44f6-9738-4edffb7a5237","Type":"ContainerStarted","Data":"a7ad494757f34363d968880ad3b7d0d4421efa7f6625cdd64568dba7adf8d25f"}
Jul 17 21:17:26.626631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.626606    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-vqd2w" event={"ID":"54dc74f1-96b7-474e-9fe5-cb1128657650","Type":"ContainerStarted","Data":"c552405b7e10e97940af6eee6dbbb955d7c9a5f74b448496f2115c8832b78789"}
Jul 17 21:17:26.628016 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.627993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-6kztn" event={"ID":"e1dda6d1-c0dd-4780-86bb-bd3f309293e6","Type":"ContainerStarted","Data":"079eeefe108182b9887e6ba055673f6f6a3c8e3cc13b51010dc5521d3d16bdd6"}
Jul 17 21:17:26.629497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.629477    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-jsttb" event={"ID":"c0620747-a811-4560-942d-1cb176ff1af5","Type":"ContainerStarted","Data":"d3d945c856eb2f2589de42479acca8e27b6827c863a91386f6114c988b8c8f58"}
Jul 17 21:17:26.630904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.630884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-dktcq" event={"ID":"51dc316d-79fd-4a8d-9f06-e44e9e84e2e5","Type":"ContainerStarted","Data":"6b99ac1fffbf4f88585f37e1d72091cb5f188f41d81003f9700396b498e985cd"}
Jul 17 21:17:26.632245 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.632220    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4" event={"ID":"8a8cf97a-2109-4a63-84b5-69bdf583dd05","Type":"ContainerStarted","Data":"590a8831afef7e7f585b91638adf01bd227bd77dd55069e4c2d11b07e31b474b"}
Jul 17 21:17:26.633480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.633456    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="aef509005021dd8640d583ac38c7e936f738d4bc1372e3d6da0583e479287ba6" exitCode=0
Jul 17 21:17:26.633559 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.633528    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"aef509005021dd8640d583ac38c7e936f738d4bc1372e3d6da0583e479287ba6"}
Jul 17 21:17:26.635559 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635544    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:17:26.635874 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635857    2546 generic.go:356] "Generic (PLEG): container finished" podID="d347b8ae-30f7-41b6-b5a9-b3adbb9f2299" containerID="b5fe0ae1cbed7259cda8fcaaf3b0eb352f666004ccad6b3d52678ce2152aa1cf" exitCode=1
Jul 17 21:17:26.635931 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635881    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"9870f29a760cb35977259e20f3dfb2eb144803c5eff8d6723a17e24bf9d5a1b2"}
Jul 17 21:17:26.635931 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635922    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"5edb0735a6e6f59481dbceeaae7333936d40a4f0022aaefe6674d0d701bfd0cd"}
Jul 17 21:17:26.636066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerDied","Data":"b5fe0ae1cbed7259cda8fcaaf3b0eb352f666004ccad6b3d52678ce2152aa1cf"}
Jul 17 21:17:26.636066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.635993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"29d68652a25f670a233848aca6b45cc154b53ff48ec657aa8253dad724f7d106"}
Jul 17 21:17:26.645553 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.645476    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9jpfn" podStartSLOduration=4.00764057 podStartE2EDuration="22.645465287s" totalImagesPullingTime="18.637824717s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:26.645428144 +0000 UTC m=+22.625798620" watchObservedRunningTime="2026-07-17 21:17:26.645465287 +0000 UTC m=+22.625835763"
Jul 17 21:17:26.694932 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.694897    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-dktcq" podStartSLOduration=4.211944078 podStartE2EDuration="22.694884495s" totalImagesPullingTime="18.482940417s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:26.665835345 +0000 UTC m=+22.646205821" watchObservedRunningTime="2026-07-17 21:17:26.694884495 +0000 UTC m=+22.675255004"
Jul 17 21:17:26.712388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.712351    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-jsttb" podStartSLOduration=4.088172723 podStartE2EDuration="22.712336593s" totalImagesPullingTime="18.62416387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:26.711699729 +0000 UTC m=+22.692070205" watchObservedRunningTime="2026-07-17 21:17:26.712336593 +0000 UTC m=+22.692707079"
Jul 17 21:17:26.731018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:26.730979    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-6kztn" podStartSLOduration=4.118735995 podStartE2EDuration="22.73096817s" totalImagesPullingTime="18.612232175s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:26.730530046 +0000 UTC m=+22.710900519" watchObservedRunningTime="2026-07-17 21:17:26.73096817 +0000 UTC m=+22.711338647"
Jul 17 21:17:27.641610 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.641573    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:17:27.642073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.642046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"2b433adde1f6ebc0ff2ec502f6be58ea245cad4e31d9d449b4af94f7cdb4481b"}
Jul 17 21:17:27.642183 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.642169    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"104883720b5dd5c19ac51b8cd824307834f6364cdeb0ee5502bc47c7fe3b28a4"}
Jul 17 21:17:27.643604 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.643573    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-tq8x7" event={"ID":"730765d7-ae92-4d45-b064-3ab24aeac23d","Type":"ContainerStarted","Data":"9b68b83cedc9553a774b48e47f4eea842441a597ee626a257a14d6e3f48cb5ef"}
Jul 17 21:17:27.664645 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.664594    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-vqd2w" podStartSLOduration=5.187205892 podStartE2EDuration="23.664578386s" totalImagesPullingTime="18.477372494s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:26.761447675 +0000 UTC m=+22.741818153" watchObservedRunningTime="2026-07-17 21:17:27.664578386 +0000 UTC m=+23.644948863"
Jul 17 21:17:27.824665 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:27.824632    2546 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 17 21:17:28.465172 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.465042    2546 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-17T21:17:27.82465287Z","UUID":"a9774894-af24-4a80-820d-2cfd028c4af4","Handler":null,"Name":"","Endpoint":""}
Jul 17 21:17:28.466575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.466542    2546 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 17 21:17:28.466732 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.466586    2546 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 17 21:17:28.532885 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.532854    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:28.533033 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:28.532971    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:28.533342 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.533321    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:28.533469 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:28.533423    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:28.533532 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.533492    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:28.533584 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:28.533560    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:28.648510 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:28.648475    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4" event={"ID":"8a8cf97a-2109-4a63-84b5-69bdf583dd05","Type":"ContainerStarted","Data":"67eae63aa5a978a8194cbac86b9a764376372db00697587726281d41de45d466"}
Jul 17 21:17:29.653379 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:29.653017    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4" event={"ID":"8a8cf97a-2109-4a63-84b5-69bdf583dd05","Type":"ContainerStarted","Data":"b8cd6453be476a9f86032d838fbc8f40e27b7153b9acf83b0668fbec93f9b4df"}
Jul 17 21:17:29.656414 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:29.656394    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:17:29.656847 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:29.656754    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"135157dc134f7c05b325d5ec202362c149677afcb84f1117321ef722898be5de"}
Jul 17 21:17:29.674989 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:29.674949    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-tq8x7" podStartSLOduration=6.957585518 podStartE2EDuration="25.674931292s" totalImagesPullingTime="18.717345774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:27.665236311 +0000 UTC m=+23.645606789" watchObservedRunningTime="2026-07-17 21:17:29.674931292 +0000 UTC m=+25.655301807"
Jul 17 21:17:29.675267 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:29.675240    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-ggqj4" podStartSLOduration=4.75112663 podStartE2EDuration="25.675228448s" totalImagesPullingTime="20.924101818s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:29.67465328 +0000 UTC m=+25.655023758" watchObservedRunningTime="2026-07-17 21:17:29.675228448 +0000 UTC m=+25.655598925"
Jul 17 21:17:30.117874 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.117841    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:30.118449 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.118430    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:30.531612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.531574    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:30.531612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.531606    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:30.531904 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:30.531709    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:30.531904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.531794    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:30.531904 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:30.531890    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:30.532066 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:30.531960    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:30.659557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.659533    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:30.660086 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:30.659784    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-jsttb"
Jul 17 21:17:31.662760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:31.662489    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="261bce07b2ef1dfd6df818396028ffffd53e8fc9716caa68a9e1c075daddec65" exitCode=0
Jul 17 21:17:31.663378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:31.662571    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"261bce07b2ef1dfd6df818396028ffffd53e8fc9716caa68a9e1c075daddec65"}
Jul 17 21:17:31.666309 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:31.666294    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:17:31.666714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:31.666691    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"e3ed735c19474f17bfc67078a5e092f149c129efed7b6e94de2d80169cf0e517"}
Jul 17 21:17:31.667028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:31.667014    2546 scope.go:117] "RemoveContainer" containerID="b5fe0ae1cbed7259cda8fcaaf3b0eb352f666004ccad6b3d52678ce2152aa1cf"
Jul 17 21:17:32.531045 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.531024    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:32.531156 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.531052    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:32.531213 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.531165    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:32.531286 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.531253    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:32.531323 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.531291    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:32.531399 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.531380    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:32.671837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.671806    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="a1dffd2fe17e96bd22f184980b28e69d9dc4aa8f4c94068df81362fb1c050aaf" exitCode=0
Jul 17 21:17:32.672263 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.671834    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"a1dffd2fe17e96bd22f184980b28e69d9dc4aa8f4c94068df81362fb1c050aaf"}
Jul 17 21:17:32.676073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.676048    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:17:32.676547 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.676522    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" event={"ID":"d347b8ae-30f7-41b6-b5a9-b3adbb9f2299","Type":"ContainerStarted","Data":"74f63c580b9f78dbbbd84c2f8077866d1d70c7c85f960b34e58a1e6d39a27cf4"}
Jul 17 21:17:32.676615 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.676585    2546 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 17 21:17:32.676878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.676859    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:32.676978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.676935    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:32.692274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.692220    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:32.692511 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.692497    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:32.897502 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.897333    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn" podStartSLOduration=10.219568933 podStartE2EDuration="28.897318947s" totalImagesPullingTime="18.677750014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:32.775240933 +0000 UTC m=+28.755611409" watchObservedRunningTime="2026-07-17 21:17:32.897318947 +0000 UTC m=+28.877689425"
Jul 17 21:17:32.897950 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.897905    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8c6r"]
Jul 17 21:17:32.898032 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.898020    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:32.898124 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.898105    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:32.913295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.913274    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qrm7f"]
Jul 17 21:17:32.913376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.913349    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:32.913421 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.913410    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:32.931470 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.931452    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwjff"]
Jul 17 21:17:32.931558 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:32.931519    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:32.931603 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:32.931588    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:33.322622 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:33.322594    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:17:33.680977 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:33.680909    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="c4dcb82a40d08a7958cbfdafb1c70e01370edb008df37cea68686a86fc13c674" exitCode=0
Jul 17 21:17:33.681385 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:33.681002    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"c4dcb82a40d08a7958cbfdafb1c70e01370edb008df37cea68686a86fc13c674"}
Jul 17 21:17:34.534076 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:34.534046    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:34.534257 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:34.534146    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:34.534257 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:34.534190    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:34.534257 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:34.534252    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:34.534578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:34.534561    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:34.534699 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:34.534663    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:36.534710 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:36.534677    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:36.535454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:36.534799    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:36.535454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:36.534810    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:36.535454 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:36.534805    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:36.535454 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:36.534895    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:36.535454 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:36.534975    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:38.161220 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.161189    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:38.161582 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.161307    2546 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:38.161582 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.161357    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret podName:aff86450-cf63-48d6-8ceb-d720e646190d nodeName:}" failed. No retries permitted until 2026-07-17 21:18:10.161342442 +0000 UTC m=+66.141712898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret") pod "global-pull-secret-syncer-mwjff" (UID: "aff86450-cf63-48d6-8ceb-d720e646190d") : object "kube-system"/"original-pull-secret" not registered
Jul 17 21:17:38.262038 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.261849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:38.262136 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.262067    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:38.262136 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.261993    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 17 21:17:38.262136 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.262122    2546 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 17 21:17:38.262136 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.262132    2546 projected.go:196] Error preparing data for projected volume kube-api-access-nrjkv for pod openshift-network-diagnostics/network-check-target-qrm7f: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:38.262257 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.262153    2546 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:38.262257 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.262176    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv podName:ec5ca31a-ad67-4400-88c4-829755a544dd nodeName:}" failed. No retries permitted until 2026-07-17 21:18:10.262162341 +0000 UTC m=+66.242532797 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-nrjkv" (UniqueName: "kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv") pod "network-check-target-qrm7f" (UID: "ec5ca31a-ad67-4400-88c4-829755a544dd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 17 21:17:38.262257 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.262195    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs podName:2cd6a154-86ef-464e-9ca0-6aff2fca8bb0 nodeName:}" failed. No retries permitted until 2026-07-17 21:18:10.262182179 +0000 UTC m=+66.242552655 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs") pod "network-metrics-daemon-p8c6r" (UID: "2cd6a154-86ef-464e-9ca0-6aff2fca8bb0") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 17 21:17:38.533877 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.533854    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:38.534029 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.533943    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mwjff" podUID="aff86450-cf63-48d6-8ceb-d720e646190d"
Jul 17 21:17:38.534029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.534023    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:38.534121 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.534103    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qrm7f" podUID="ec5ca31a-ad67-4400-88c4-829755a544dd"
Jul 17 21:17:38.534156 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.534147    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:38.534218 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:38.534206    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-p8c6r" podUID="2cd6a154-86ef-464e-9ca0-6aff2fca8bb0"
Jul 17 21:17:38.693589 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.693565    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="24d3a555e8267503fd997b970da7e9efd1e37b5a3d330f5dbef760b5e9eada5c" exitCode=0
Jul 17 21:17:38.693678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.693620    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"24d3a555e8267503fd997b970da7e9efd1e37b5a3d330f5dbef760b5e9eada5c"}
Jul 17 21:17:38.906494 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.906475    2546 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-135-220.ec2.internal" event="NodeReady"
Jul 17 21:17:38.906631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.906603    2546 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 17 21:17:38.969408 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.969380    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-66784678fd-r72dv"]
Jul 17 21:17:38.981193 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.981169    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zsxq7"]
Jul 17 21:17:38.981291 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.981280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:38.984300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.984285    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 17 21:17:38.984391 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.984372    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 17 21:17:38.984440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.984389    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fr9hx\""
Jul 17 21:17:38.984631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.984612    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 17 21:17:38.989821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.989805    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 17 21:17:38.993502 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.993483    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-jhp4h"]
Jul 17 21:17:38.993619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.993603    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:38.996830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.996802    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 17 21:17:38.996830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.996822    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwcbh\""
Jul 17 21:17:38.997163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:38.997139    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 17 21:17:39.003665 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.003650    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-66784678fd-r72dv"]
Jul 17 21:17:39.003774 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.003680    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zsxq7"]
Jul 17 21:17:39.003774 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.003766    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.008204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.008185    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7vktg\""
Jul 17 21:17:39.008296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.008262    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 17 21:17:39.008296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.008273    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 17 21:17:39.008402 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.008284    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 17 21:17:39.012113 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.012094    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhp4h"]
Jul 17 21:17:39.066865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066822    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-tmp-dir\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.066865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066847    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.066865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066870    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066891    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-config-volume\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.067015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066931    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfqrl\" (UniqueName: \"kubernetes.io/projected/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-kube-api-access-bfqrl\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.067015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066960    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.066977    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.067044    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.067061    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.067096    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.067134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.067123    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6llx\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.067247 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.067172    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.167750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.167694    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.167761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.167807    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.167830    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6llx\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.167854    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.167874    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.167925    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:17:39.667910169 +0000 UTC m=+35.648280623 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.167933    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.167979    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.167997    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:17:39.667979907 +0000 UTC m=+35.648350363 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168028    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9628a26-4ddd-46b2-8a74-39731b45c70c-cert\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168072    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5bjt\" (UniqueName: \"kubernetes.io/projected/a9628a26-4ddd-46b2-8a74-39731b45c70c-kube-api-access-k5bjt\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168105    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-tmp-dir\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168132    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168167    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168203    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-config-volume\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168215    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168226    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfqrl\" (UniqueName: \"kubernetes.io/projected/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-kube-api-access-bfqrl\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168263    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168287    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.168832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168819    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-tmp-dir\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.169092 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.168884    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.169169 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.169131    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-config-volume\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.169824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.169740    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.174618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.174592    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.174845 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.174826    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.177389 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.177369    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6llx\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.177500 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.177372    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.183850 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.183829    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfqrl\" (UniqueName: \"kubernetes.io/projected/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-kube-api-access-bfqrl\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.270473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.270447    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5bjt\" (UniqueName: \"kubernetes.io/projected/a9628a26-4ddd-46b2-8a74-39731b45c70c-kube-api-access-k5bjt\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.270570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.270557    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9628a26-4ddd-46b2-8a74-39731b45c70c-cert\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.272626 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.272610    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a9628a26-4ddd-46b2-8a74-39731b45c70c-cert\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.279711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.279691    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5bjt\" (UniqueName: \"kubernetes.io/projected/a9628a26-4ddd-46b2-8a74-39731b45c70c-kube-api-access-k5bjt\") pod \"ingress-canary-jhp4h\" (UID: \"a9628a26-4ddd-46b2-8a74-39731b45c70c\") " pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.310545 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.310524    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-jhp4h"
Jul 17 21:17:39.461787 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.461763    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-jhp4h"]
Jul 17 21:17:39.465623 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:39.465598    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9628a26_4ddd_46b2_8a74_39731b45c70c.slice/crio-a06ab9939639b174a873aa7174cb8a0cc0ceeecbb20415121324fc9616676457 WatchSource:0}: Error finding container a06ab9939639b174a873aa7174cb8a0cc0ceeecbb20415121324fc9616676457: Status 404 returned error can't find the container with id a06ab9939639b174a873aa7174cb8a0cc0ceeecbb20415121324fc9616676457
Jul 17 21:17:39.674446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.674389    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:39.674446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.674428    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:39.674617 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.674539    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:39.674617 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.674555    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:39.674617 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.674575    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:39.674617 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.674586    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:17:40.674571671 +0000 UTC m=+36.654942126 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:39.674790 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:39.674640    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:17:40.674621927 +0000 UTC m=+36.654992399 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:17:39.696569 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.696546    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhp4h" event={"ID":"a9628a26-4ddd-46b2-8a74-39731b45c70c","Type":"ContainerStarted","Data":"a06ab9939639b174a873aa7174cb8a0cc0ceeecbb20415121324fc9616676457"}
Jul 17 21:17:39.699651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.699631    2546 generic.go:356] "Generic (PLEG): container finished" podID="b196a3d2-d251-432c-bcfd-abb6171d087b" containerID="441b6fb206505f5c509630aaaf33a139d721ff73c6686c79cbee9997c26251f5" exitCode=0
Jul 17 21:17:39.699762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:39.699658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerDied","Data":"441b6fb206505f5c509630aaaf33a139d721ff73c6686c79cbee9997c26251f5"}
Jul 17 21:17:40.534324 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.534295    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:17:40.534873 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.534397    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:17:40.534873 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.534426    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:17:40.537764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537739    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzlv\""
Jul 17 21:17:40.537764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537751    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:17:40.537764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537761    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lpmh2\""
Jul 17 21:17:40.538008 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537789    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:17:40.538008 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537805    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:17:40.538008 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.537734    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:17:40.682144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.682049    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:40.682275 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.682189    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:40.682275 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:40.682212    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:40.682275 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:40.682228    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:40.682444 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:40.682284    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:17:42.68226554 +0000 UTC m=+38.662636015 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:17:40.682444 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:40.682293    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:40.682444 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:40.682400    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:17:42.682384177 +0000 UTC m=+38.662754639 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:40.705904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.705881    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" event={"ID":"b196a3d2-d251-432c-bcfd-abb6171d087b","Type":"ContainerStarted","Data":"9f27eaab31db2fe030b38cd27b11f278793f2d390c9a60b0a8a9af23f298ccae"}
Jul 17 21:17:40.733791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:40.733741    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-9cgfc" podStartSLOduration=8.063492359 podStartE2EDuration="36.733712633s" totalImagesPullingTime="28.670220274s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:40.731833115 +0000 UTC m=+36.712203592" watchObservedRunningTime="2026-07-17 21:17:40.733712633 +0000 UTC m=+36.714083108"
Jul 17 21:17:42.696525 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:42.696455    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:42.696525 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:42.696498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:42.697043 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:42.696606    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:42.697043 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:42.696607    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:42.697043 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:42.696625    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:42.697043 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:42.696655    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:17:46.696642754 +0000 UTC m=+42.677013210 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:42.697043 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:42.696670    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:17:46.696662484 +0000 UTC m=+42.677032939 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:17:42.712073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:42.712051    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-jhp4h" event={"ID":"a9628a26-4ddd-46b2-8a74-39731b45c70c","Type":"ContainerStarted","Data":"36a87feff0cb668f8f67a780edcc7a60c4feb10358a4a625a120acf585b19e47"}
Jul 17 21:17:42.733040 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:42.732996    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-jhp4h" podStartSLOduration=2.270909934 podStartE2EDuration="4.732981245s" totalImagesPullingTime="2.462071311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:42.731836247 +0000 UTC m=+38.712206725" watchObservedRunningTime="2026-07-17 21:17:42.732981245 +0000 UTC m=+38.713351723"
Jul 17 21:17:44.875906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.875874    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-rxq8w"]
Jul 17 21:17:44.907911 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.907886    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxq8w"]
Jul 17 21:17:44.908026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.907976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:44.910343 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.910321    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 17 21:17:44.910646 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.910627    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 17 21:17:44.911268 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.911248    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-rvj24\""
Jul 17 21:17:44.911375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.911246    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 17 21:17:44.911375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:44.911254    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 17 21:17:45.011080 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.011051    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/323ca98f-d27a-455c-9dea-22cfd7e88e9e-data-volume\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.011187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.011098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dd74\" (UniqueName: \"kubernetes.io/projected/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-api-access-6dd74\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.011187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.011157    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/323ca98f-d27a-455c-9dea-22cfd7e88e9e-crio-socket\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.011303 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.011209    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.011303 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.011240    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111558 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111535    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/323ca98f-d27a-455c-9dea-22cfd7e88e9e-data-volume\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111642 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111569    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dd74\" (UniqueName: \"kubernetes.io/projected/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-api-access-6dd74\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111642 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111620    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/323ca98f-d27a-455c-9dea-22cfd7e88e9e-crio-socket\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111650    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111667    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111854 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.111763    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:17:45.111854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/323ca98f-d27a-455c-9dea-22cfd7e88e9e-crio-socket\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.111830    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/323ca98f-d27a-455c-9dea-22cfd7e88e9e-data-volume\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.111854 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.111823    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:45.611806975 +0000 UTC m=+41.592177429 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : configmap "kube-rbac-proxy" not found
Jul 17 21:17:45.112004 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.111835    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:45.112004 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.111903    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:45.611890725 +0000 UTC m=+41.592261180 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:45.146167 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.146117    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dd74\" (UniqueName: \"kubernetes.io/projected/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-api-access-6dd74\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.151369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.151350    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"]
Jul 17 21:17:45.169213 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.169193    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"]
Jul 17 21:17:45.169296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.169277    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"
Jul 17 21:17:45.173901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.173882    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 17 21:17:45.173994 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.173886    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 17 21:17:45.173994 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.173911    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-rw64w\""
Jul 17 21:17:45.313052 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.313023    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/f4cb5585-d50e-4c20-9573-8ba754f22652-kube-api-access-8rmbh\") pod \"migrator-7cc9c7ffd7-8jv2q\" (UID: \"f4cb5585-d50e-4c20-9573-8ba754f22652\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"
Jul 17 21:17:45.413388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.413343    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/f4cb5585-d50e-4c20-9573-8ba754f22652-kube-api-access-8rmbh\") pod \"migrator-7cc9c7ffd7-8jv2q\" (UID: \"f4cb5585-d50e-4c20-9573-8ba754f22652\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"
Jul 17 21:17:45.422894 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.422876    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rmbh\" (UniqueName: \"kubernetes.io/projected/f4cb5585-d50e-4c20-9573-8ba754f22652-kube-api-access-8rmbh\") pod \"migrator-7cc9c7ffd7-8jv2q\" (UID: \"f4cb5585-d50e-4c20-9573-8ba754f22652\") " pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"
Jul 17 21:17:45.478032 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.478013    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"
Jul 17 21:17:45.615555 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.615521    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.615682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.615566    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:45.615682 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.615646    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:17:45.615682 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.615655    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:45.615828 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.615699    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:46.61568449 +0000 UTC m=+42.596054945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : configmap "kube-rbac-proxy" not found
Jul 17 21:17:45.615828 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:45.615712    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:46.615706564 +0000 UTC m=+42.596077018 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:45.636917 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.636892    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q"]
Jul 17 21:17:45.640095 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:45.640067    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4cb5585_d50e_4c20_9573_8ba754f22652.slice/crio-29c62727f60c8d064c52066a07c3a20ee23bd1bcf62a8a2cc53546d22fbb9f82 WatchSource:0}: Error finding container 29c62727f60c8d064c52066a07c3a20ee23bd1bcf62a8a2cc53546d22fbb9f82: Status 404 returned error can't find the container with id 29c62727f60c8d064c52066a07c3a20ee23bd1bcf62a8a2cc53546d22fbb9f82
Jul 17 21:17:45.720027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:45.719972    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q" event={"ID":"f4cb5585-d50e-4c20-9573-8ba754f22652","Type":"ContainerStarted","Data":"29c62727f60c8d064c52066a07c3a20ee23bd1bcf62a8a2cc53546d22fbb9f82"}
Jul 17 21:17:46.459235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:46.459193    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dktcq_51dc316d-79fd-4a8d-9f06-e44e9e84e2e5/dns-node-resolver/0.log"
Jul 17 21:17:46.624314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:46.624283    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:46.624469 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:46.624319    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:46.624469 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.624431    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:17:46.624469 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.624450    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:46.624569 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.624484    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:48.624472044 +0000 UTC m=+44.604842499 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : configmap "kube-rbac-proxy" not found
Jul 17 21:17:46.624569 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.624516    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:48.624489775 +0000 UTC m=+44.604860230 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:46.724997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:46.724908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:46.724997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:46.724953    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:46.725181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.725052    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:46.725181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.725056    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:46.725181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.725073    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:46.725181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.725107    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:17:54.72509116 +0000 UTC m=+50.705461614 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:46.725181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:46.725122    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:17:54.725116093 +0000 UTC m=+50.705486548 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:17:47.582215 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.582187    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-m42qg"]
Jul 17 21:17:47.619578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.619549    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-m42qg"]
Jul 17 21:17:47.619687 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.619640    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.626536 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.626513    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 17 21:17:47.626838 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.626819    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 17 21:17:47.627553 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.627537    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 17 21:17:47.627645 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.627549    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 17 21:17:47.627645 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.627554    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fkkvl\""
Jul 17 21:17:47.674986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.674940    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqd2w_54dc74f1-96b7-474e-9fe5-cb1128657650/node-ca/0.log"
Jul 17 21:17:47.728684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.728651    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q" event={"ID":"f4cb5585-d50e-4c20-9573-8ba754f22652","Type":"ContainerStarted","Data":"26acbc1145398c0fd54d55d7ffb679f4f7f19bfffe1aaf3bf02dcd9bfa13755f"}
Jul 17 21:17:47.728854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.728753    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q" event={"ID":"f4cb5585-d50e-4c20-9573-8ba754f22652","Type":"ContainerStarted","Data":"2649965abfb0469e111a37ecd0b84ae3a6a075696b1efdc9137b9ea026cd6cfc"}
Jul 17 21:17:47.732264 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.732243    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-key\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.732324 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.732273    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mq2dq\" (UniqueName: \"kubernetes.io/projected/7a8a08cb-7e48-4420-a31f-9ac0caab9482-kube-api-access-mq2dq\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.732358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.732351    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-cabundle\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.758659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.758613    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7cc9c7ffd7-8jv2q" podStartSLOduration=0.991227942 podStartE2EDuration="2.758600129s" totalImagesPullingTime="1.767372187s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:47.758449345 +0000 UTC m=+43.738819826" watchObservedRunningTime="2026-07-17 21:17:47.758600129 +0000 UTC m=+43.738970605"
Jul 17 21:17:47.833541 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.833494    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-cabundle\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.833708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.833607    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-key\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.833708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.833633    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mq2dq\" (UniqueName: \"kubernetes.io/projected/7a8a08cb-7e48-4420-a31f-9ac0caab9482-kube-api-access-mq2dq\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.834355 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.834326    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-cabundle\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.836157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.836137    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7a8a08cb-7e48-4420-a31f-9ac0caab9482-signing-key\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.848965 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.848937    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mq2dq\" (UniqueName: \"kubernetes.io/projected/7a8a08cb-7e48-4420-a31f-9ac0caab9482-kube-api-access-mq2dq\") pod \"service-ca-58598695f7-m42qg\" (UID: \"7a8a08cb-7e48-4420-a31f-9ac0caab9482\") " pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:47.932212 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:47.932123    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-m42qg"
Jul 17 21:17:48.110630 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:48.110599    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-m42qg"]
Jul 17 21:17:48.113820 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:17:48.113795    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a8a08cb_7e48_4420_a31f_9ac0caab9482.slice/crio-b6637d61acaa09144c96f6ca2f1c83a7d8da9dbb85bc4b915c52641421279c43 WatchSource:0}: Error finding container b6637d61acaa09144c96f6ca2f1c83a7d8da9dbb85bc4b915c52641421279c43: Status 404 returned error can't find the container with id b6637d61acaa09144c96f6ca2f1c83a7d8da9dbb85bc4b915c52641421279c43
Jul 17 21:17:48.261823 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:48.261783    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhp4h_a9628a26-4ddd-46b2-8a74-39731b45c70c/serve-healthcheck-canary/0.log"
Jul 17 21:17:48.640320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:48.640237    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:48.640320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:48.640271    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:48.640693 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:48.640358    2546 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 17 21:17:48.640693 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:48.640397    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:48.640693 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:48.640407    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:52.640393331 +0000 UTC m=+48.620763787 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : configmap "kube-rbac-proxy" not found
Jul 17 21:17:48.640693 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:48.640547    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:17:52.640531099 +0000 UTC m=+48.620901555 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:48.732029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:48.731995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-m42qg" event={"ID":"7a8a08cb-7e48-4420-a31f-9ac0caab9482","Type":"ContainerStarted","Data":"b6637d61acaa09144c96f6ca2f1c83a7d8da9dbb85bc4b915c52641421279c43"}
Jul 17 21:17:50.738239 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:50.738212    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-m42qg" event={"ID":"7a8a08cb-7e48-4420-a31f-9ac0caab9482","Type":"ContainerStarted","Data":"3da82c72d21559e1fe71e43b54c8997adc6c523ccecac89145ec4ea4129c865e"}
Jul 17 21:17:51.764371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:51.764320    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-m42qg" podStartSLOduration=2.2465644 podStartE2EDuration="4.764302918s" totalImagesPullingTime="2.517738518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:17:51.763714444 +0000 UTC m=+47.744084922" watchObservedRunningTime="2026-07-17 21:17:51.764302918 +0000 UTC m=+47.744673373"
Jul 17 21:17:52.670798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:52.670759    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:52.670798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:52.670811    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:52.671059 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:52.670925    2546 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:52.671059 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:52.671013    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls podName:323ca98f-d27a-455c-9dea-22cfd7e88e9e nodeName:}" failed. No retries permitted until 2026-07-17 21:18:00.670988342 +0000 UTC m=+56.651358802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-rxq8w" (UID: "323ca98f-d27a-455c-9dea-22cfd7e88e9e") : secret "insights-runtime-extractor-tls" not found
Jul 17 21:17:52.671420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:52.671400    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/323ca98f-d27a-455c-9dea-22cfd7e88e9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:17:54.787280 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:54.787245    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:17:54.787280 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:17:54.787286    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:17:54.787792 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:54.787380    2546 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 17 21:17:54.787792 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:54.787383    2546 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 17 21:17:54.787792 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:54.787397    2546 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-66784678fd-r72dv: secret "image-registry-tls" not found
Jul 17 21:17:54.787792 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:54.787424    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls podName:e6c349f1-c206-49d8-a9df-8ab81a3b68ea nodeName:}" failed. No retries permitted until 2026-07-17 21:18:10.787411681 +0000 UTC m=+66.767782136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls") pod "dns-default-zsxq7" (UID: "e6c349f1-c206-49d8-a9df-8ab81a3b68ea") : secret "dns-default-metrics-tls" not found
Jul 17 21:17:54.787792 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:17:54.787437    2546 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls podName:00481bd8-0cf6-4500-9018-77df9044be6c nodeName:}" failed. No retries permitted until 2026-07-17 21:18:10.787430256 +0000 UTC m=+66.767800711 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls") pod "image-registry-66784678fd-r72dv" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c") : secret "image-registry-tls" not found
Jul 17 21:18:00.736808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:00.736756    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:18:00.739241 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:00.739219    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/323ca98f-d27a-455c-9dea-22cfd7e88e9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-rxq8w\" (UID: \"323ca98f-d27a-455c-9dea-22cfd7e88e9e\") " pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:18:00.819076 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:00.819036    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-rxq8w"
Jul 17 21:18:00.950046 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:00.950010    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-rxq8w"]
Jul 17 21:18:00.953063 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:00.953035    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod323ca98f_d27a_455c_9dea_22cfd7e88e9e.slice/crio-156c6b5fae73017bc78a7a4cc8c963522e520b1658bd1609272272a617ed00e8 WatchSource:0}: Error finding container 156c6b5fae73017bc78a7a4cc8c963522e520b1658bd1609272272a617ed00e8: Status 404 returned error can't find the container with id 156c6b5fae73017bc78a7a4cc8c963522e520b1658bd1609272272a617ed00e8
Jul 17 21:18:01.767215 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:01.767166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxq8w" event={"ID":"323ca98f-d27a-455c-9dea-22cfd7e88e9e","Type":"ContainerStarted","Data":"156c6b5fae73017bc78a7a4cc8c963522e520b1658bd1609272272a617ed00e8"}
Jul 17 21:18:04.696618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:04.696584    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-49bsn"
Jul 17 21:18:06.785248 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:06.785201    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxq8w" event={"ID":"323ca98f-d27a-455c-9dea-22cfd7e88e9e","Type":"ContainerStarted","Data":"a644e1fb348f974a8cd431fbca8976005056cbb9fe89d77d5fd93de02c33791f"}
Jul 17 21:18:09.413733 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.413678    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"]
Jul 17 21:18:09.435030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.434996    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"]
Jul 17 21:18:09.435214 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.435158    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.437378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.437352    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 17 21:18:09.437923 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.437897    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 17 21:18:09.438156 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.438123    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 17 21:18:09.438234 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.438164    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 17 21:18:09.456669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.456638    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"]
Jul 17 21:18:09.456857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.456694    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"]
Jul 17 21:18:09.456857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.456712    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66784678fd-r72dv"]
Jul 17 21:18:09.456857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.456768    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.457022 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:18:09.456941    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-66784678fd-r72dv" podUID="00481bd8-0cf6-4500-9018-77df9044be6c"
Jul 17 21:18:09.459320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.459293    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-cnx5q\""
Jul 17 21:18:09.459601 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.459585    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 17 21:18:09.508773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.508738    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-549479f6c5-v5cqb"]
Jul 17 21:18:09.512890 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.512865    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc59b820-a70f-43fb-81ee-d818b67c5c38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.513021 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.512983    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-klusterlet-config\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.513067 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.513035    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-tmp\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.513104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.513079    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jt6xj\" (UniqueName: \"kubernetes.io/projected/dc59b820-a70f-43fb-81ee-d818b67c5c38-kube-api-access-jt6xj\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.513160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.513116    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj4gn\" (UniqueName: \"kubernetes.io/projected/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-kube-api-access-sj4gn\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.532217 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.532193    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.536196 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.536164    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549479f6c5-v5cqb"]
Jul 17 21:18:09.613779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613741    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc59b820-a70f-43fb-81ee-d818b67c5c38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.613978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613785    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbxqg\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-kube-api-access-vbxqg\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.613978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613819    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-bound-sa-token\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.613978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613861    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-klusterlet-config\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.613978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613940    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-installation-pull-secrets\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.613978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613973    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-tmp\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.613995    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jt6xj\" (UniqueName: \"kubernetes.io/projected/dc59b820-a70f-43fb-81ee-d818b67c5c38-kube-api-access-jt6xj\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614029    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj4gn\" (UniqueName: \"kubernetes.io/projected/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-kube-api-access-sj4gn\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614071    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7d800b3-023c-498b-af08-9ec4308ea109-ca-trust-extracted\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614095    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-trusted-ca\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614141    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-image-registry-private-configuration\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614177    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-registry-certificates\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.614242 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614204    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-registry-tls\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.614489 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.614352    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-tmp\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.616569 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.616537    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-klusterlet-config\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.616669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.616545    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/dc59b820-a70f-43fb-81ee-d818b67c5c38-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.623602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.623581    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jt6xj\" (UniqueName: \"kubernetes.io/projected/dc59b820-a70f-43fb-81ee-d818b67c5c38-kube-api-access-jt6xj\") pod \"managed-serviceaccount-addon-agent-897466b88-5xmfw\" (UID: \"dc59b820-a70f-43fb-81ee-d818b67c5c38\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.631311 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.631291    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj4gn\" (UniqueName: \"kubernetes.io/projected/52717cb2-c3ef-4d59-85ce-a7867fc8ba24-kube-api-access-sj4gn\") pod \"klusterlet-addon-workmgr-58c899dfc9-mjgv2\" (UID: \"52717cb2-c3ef-4d59-85ce-a7867fc8ba24\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.714624 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714544    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7d800b3-023c-498b-af08-9ec4308ea109-ca-trust-extracted\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.714624 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714581    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-trusted-ca\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.714883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714715    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-image-registry-private-configuration\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.714883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-registry-certificates\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.714883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714829    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-registry-tls\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.714883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714865    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbxqg\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-kube-api-access-vbxqg\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.715073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714906    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-bound-sa-token\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.715073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.714948    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b7d800b3-023c-498b-af08-9ec4308ea109-ca-trust-extracted\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.715073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.715006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-installation-pull-secrets\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.715588 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.715561    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-trusted-ca\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.720079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.716889    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b7d800b3-023c-498b-af08-9ec4308ea109-registry-certificates\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.720795 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.720774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-installation-pull-secrets\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.720892 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.720877    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b7d800b3-023c-498b-af08-9ec4308ea109-image-registry-private-configuration\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.720961 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.720940    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-registry-tls\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.730197 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.730164    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-bound-sa-token\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.730289 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.730269    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbxqg\" (UniqueName: \"kubernetes.io/projected/b7d800b3-023c-498b-af08-9ec4308ea109-kube-api-access-vbxqg\") pod \"image-registry-549479f6c5-v5cqb\" (UID: \"b7d800b3-023c-498b-af08-9ec4308ea109\") " pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.746586 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.746559    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:09.781738 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.781693    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"
Jul 17 21:18:09.796504 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.796482    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:18:09.824375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.824352    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:18:09.850847 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.850815    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-fr9hx\""
Jul 17 21:18:09.852879 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.852861    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:09.916360 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916323    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916374    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916396    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916440    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916461    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6llx\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916489    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916539    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:09.916971 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916842    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:18:09.916971 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916955    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:18:09.917069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.916971    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:18:09.918780 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.918756    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:18:09.918878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.918787    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:18:09.919028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.919005    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:09.919135 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:09.919065    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx" (OuterVolumeSpecName: "kube-api-access-z6llx") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "kube-api-access-z6llx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018051    2546 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-image-registry-private-configuration\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018082    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-trusted-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018101    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6llx\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-kube-api-access-z6llx\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018114    2546 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/00481bd8-0cf6-4500-9018-77df9044be6c-ca-trust-extracted\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018128    2546 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/00481bd8-0cf6-4500-9018-77df9044be6c-installation-pull-secrets\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018140    2546 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-bound-sa-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.018496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.018153    2546 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/00481bd8-0cf6-4500-9018-77df9044be6c-registry-certificates\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:10.115929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.115898    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw"]
Jul 17 21:18:10.118640 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.118610    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddc59b820_a70f_43fb_81ee_d818b67c5c38.slice/crio-15988adb8dca015b296e75ff0cb3594e71af51ef8ccc455ee74400025488e6b8 WatchSource:0}: Error finding container 15988adb8dca015b296e75ff0cb3594e71af51ef8ccc455ee74400025488e6b8: Status 404 returned error can't find the container with id 15988adb8dca015b296e75ff0cb3594e71af51ef8ccc455ee74400025488e6b8
Jul 17 21:18:10.219296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.219260    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:18:10.224276 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.224222    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 17 21:18:10.232518 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.232498    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/aff86450-cf63-48d6-8ceb-d720e646190d-original-pull-secret\") pod \"global-pull-secret-syncer-mwjff\" (UID: \"aff86450-cf63-48d6-8ceb-d720e646190d\") " pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:18:10.245632 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.245606    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mwjff"
Jul 17 21:18:10.320533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.320498    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:18:10.320839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.320570    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:18:10.323793 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.323764    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 17 21:18:10.323793 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.323786    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 17 21:18:10.333861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.333835    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cd6a154-86ef-464e-9ca0-6aff2fca8bb0-metrics-certs\") pod \"network-metrics-daemon-p8c6r\" (UID: \"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0\") " pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:18:10.338852 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.338831    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 17 21:18:10.340917 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.340894    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"]
Jul 17 21:18:10.344711 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.344681    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52717cb2_c3ef_4d59_85ce_a7867fc8ba24.slice/crio-aa97d2e41c6ca0231388c7360d9436469dc79cb6d95a592b6e8584ad5e9bf597 WatchSource:0}: Error finding container aa97d2e41c6ca0231388c7360d9436469dc79cb6d95a592b6e8584ad5e9bf597: Status 404 returned error can't find the container with id aa97d2e41c6ca0231388c7360d9436469dc79cb6d95a592b6e8584ad5e9bf597
Jul 17 21:18:10.344877 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.344860    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrjkv\" (UniqueName: \"kubernetes.io/projected/ec5ca31a-ad67-4400-88c4-829755a544dd-kube-api-access-nrjkv\") pod \"network-check-target-qrm7f\" (UID: \"ec5ca31a-ad67-4400-88c4-829755a544dd\") " pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:18:10.345404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.345338    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-549479f6c5-v5cqb"]
Jul 17 21:18:10.350564 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.350536    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7d800b3_023c_498b_af08_9ec4308ea109.slice/crio-609150b848a7806ccbe5516ffa273452b2f4f7cc58c3a8b538ebd0a8e25a303d WatchSource:0}: Error finding container 609150b848a7806ccbe5516ffa273452b2f4f7cc58c3a8b538ebd0a8e25a303d: Status 404 returned error can't find the container with id 609150b848a7806ccbe5516ffa273452b2f4f7cc58c3a8b538ebd0a8e25a303d
Jul 17 21:18:10.402406 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.402378    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mwjff"]
Jul 17 21:18:10.406933 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.406908    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff86450_cf63_48d6_8ceb_d720e646190d.slice/crio-a30eb67e3cf1f9e1ec96a1af20836888e37a86703c15577575069ec7ff9cf71f WatchSource:0}: Error finding container a30eb67e3cf1f9e1ec96a1af20836888e37a86703c15577575069ec7ff9cf71f: Status 404 returned error can't find the container with id a30eb67e3cf1f9e1ec96a1af20836888e37a86703c15577575069ec7ff9cf71f
Jul 17 21:18:10.555582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.555520    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-lpmh2\""
Jul 17 21:18:10.559951 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.559917    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzlv\""
Jul 17 21:18:10.562947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.562923    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:18:10.568135 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.568111    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-p8c6r"
Jul 17 21:18:10.757896 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.757865    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qrm7f"]
Jul 17 21:18:10.762068 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.762035    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec5ca31a_ad67_4400_88c4_829755a544dd.slice/crio-d4bf54d89e891cf665362484913375eb56f214653bbcf64957a3ef9af227bf51 WatchSource:0}: Error finding container d4bf54d89e891cf665362484913375eb56f214653bbcf64957a3ef9af227bf51: Status 404 returned error can't find the container with id d4bf54d89e891cf665362484913375eb56f214653bbcf64957a3ef9af227bf51
Jul 17 21:18:10.800189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.800109    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-p8c6r"]
Jul 17 21:18:10.804430 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:10.804398    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd6a154_86ef_464e_9ca0_6aff2fca8bb0.slice/crio-28b4a16e84d8452ca9884a443b9635abc03e7c9326e3b4373fea473b8ea9fb68 WatchSource:0}: Error finding container 28b4a16e84d8452ca9884a443b9635abc03e7c9326e3b4373fea473b8ea9fb68: Status 404 returned error can't find the container with id 28b4a16e84d8452ca9884a443b9635abc03e7c9326e3b4373fea473b8ea9fb68
Jul 17 21:18:10.806007 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.805875    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw" event={"ID":"dc59b820-a70f-43fb-81ee-d818b67c5c38","Type":"ContainerStarted","Data":"15988adb8dca015b296e75ff0cb3594e71af51ef8ccc455ee74400025488e6b8"}
Jul 17 21:18:10.811030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.810999    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxq8w" event={"ID":"323ca98f-d27a-455c-9dea-22cfd7e88e9e","Type":"ContainerStarted","Data":"c9e87a96a04cfdc09454cea6918e0a4cbbafee8e59f35fc193242a61fb021a96"}
Jul 17 21:18:10.812486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.812457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qrm7f" event={"ID":"ec5ca31a-ad67-4400-88c4-829755a544dd","Type":"ContainerStarted","Data":"d4bf54d89e891cf665362484913375eb56f214653bbcf64957a3ef9af227bf51"}
Jul 17 21:18:10.813886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.813857    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mwjff" event={"ID":"aff86450-cf63-48d6-8ceb-d720e646190d","Type":"ContainerStarted","Data":"a30eb67e3cf1f9e1ec96a1af20836888e37a86703c15577575069ec7ff9cf71f"}
Jul 17 21:18:10.815506 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.815476    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb" event={"ID":"b7d800b3-023c-498b-af08-9ec4308ea109","Type":"ContainerStarted","Data":"147e34c537ba9d9ea5e206f9a6a8dbad5e70e292934d76a66f67a652d1be9bdf"}
Jul 17 21:18:10.815606 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.815525    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb" event={"ID":"b7d800b3-023c-498b-af08-9ec4308ea109","Type":"ContainerStarted","Data":"609150b848a7806ccbe5516ffa273452b2f4f7cc58c3a8b538ebd0a8e25a303d"}
Jul 17 21:18:10.815606 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.815558    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:10.816991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.816949    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2" event={"ID":"52717cb2-c3ef-4d59-85ce-a7867fc8ba24","Type":"ContainerStarted","Data":"aa97d2e41c6ca0231388c7360d9436469dc79cb6d95a592b6e8584ad5e9bf597"}
Jul 17 21:18:10.817092 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.816969    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:18:10.826249 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.826219    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:18:10.826373 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.826284    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:18:10.829088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.829068    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e6c349f1-c206-49d8-a9df-8ab81a3b68ea-metrics-tls\") pod \"dns-default-zsxq7\" (UID: \"e6c349f1-c206-49d8-a9df-8ab81a3b68ea\") " pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:18:10.829186 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.829070    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"image-registry-66784678fd-r72dv\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") " pod="openshift-image-registry/image-registry-66784678fd-r72dv"
Jul 17 21:18:10.863066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.863009    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb" podStartSLOduration=1.862991729 podStartE2EDuration="1.862991729s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:10.859802025 +0000 UTC m=+66.840172501" watchObservedRunningTime="2026-07-17 21:18:10.862991729 +0000 UTC m=+66.843362184"
Jul 17 21:18:10.926631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.926584    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-66784678fd-r72dv"]
Jul 17 21:18:10.926837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.926789    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") pod \"00481bd8-0cf6-4500-9018-77df9044be6c\" (UID: \"00481bd8-0cf6-4500-9018-77df9044be6c\") "
Jul 17 21:18:10.930531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.930487    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "00481bd8-0cf6-4500-9018-77df9044be6c" (UID: "00481bd8-0cf6-4500-9018-77df9044be6c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:18:10.938404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:10.938376    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-66784678fd-r72dv"]
Jul 17 21:18:11.028793 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.028750    2546 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/00481bd8-0cf6-4500-9018-77df9044be6c-registry-tls\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:18:11.104923 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.104843    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-lwcbh\""
Jul 17 21:18:11.113022 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.112991    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:18:11.284569 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.284533    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zsxq7"]
Jul 17 21:18:11.290707 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:11.290662    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6c349f1_c206_49d8_a9df_8ab81a3b68ea.slice/crio-d07030bc8c354a3110d10adb84a1100cc3ac9ba44dc00b4951126102fceec4b8 WatchSource:0}: Error finding container d07030bc8c354a3110d10adb84a1100cc3ac9ba44dc00b4951126102fceec4b8: Status 404 returned error can't find the container with id d07030bc8c354a3110d10adb84a1100cc3ac9ba44dc00b4951126102fceec4b8
Jul 17 21:18:11.828038 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.826319    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8c6r" event={"ID":"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0","Type":"ContainerStarted","Data":"28b4a16e84d8452ca9884a443b9635abc03e7c9326e3b4373fea473b8ea9fb68"}
Jul 17 21:18:11.828526 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:11.828189    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsxq7" event={"ID":"e6c349f1-c206-49d8-a9df-8ab81a3b68ea","Type":"ContainerStarted","Data":"d07030bc8c354a3110d10adb84a1100cc3ac9ba44dc00b4951126102fceec4b8"}
Jul 17 21:18:12.536980 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:12.536698    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00481bd8-0cf6-4500-9018-77df9044be6c" path="/var/lib/kubelet/pods/00481bd8-0cf6-4500-9018-77df9044be6c/volumes"
Jul 17 21:18:12.837103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:12.836820    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8c6r" event={"ID":"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0","Type":"ContainerStarted","Data":"d6cd1a65e5fa9a4ded2d63166ac37b01855637abd595b6d8f0beb657dccbb762"}
Jul 17 21:18:12.837103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:12.836884    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-p8c6r" event={"ID":"2cd6a154-86ef-464e-9ca0-6aff2fca8bb0","Type":"ContainerStarted","Data":"9ad23263a3b003651757570db861e0dcec4d5961b7115bf4482d18bbc806e80a"}
Jul 17 21:18:12.859144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:12.858567    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-p8c6r" podStartSLOduration=67.481063931 podStartE2EDuration="1m8.858543636s" totalImagesPullingTime="1.377479705s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:12.856846321 +0000 UTC m=+68.837216799" watchObservedRunningTime="2026-07-17 21:18:12.858543636 +0000 UTC m=+68.838914114"
Jul 17 21:18:17.859587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.859546    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mwjff" event={"ID":"aff86450-cf63-48d6-8ceb-d720e646190d","Type":"ContainerStarted","Data":"b08a1421817322ccef7d978e2ea352a76cc1c9396ec2f8214f9958169c616fa0"}
Jul 17 21:18:17.863678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.863039    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsxq7" event={"ID":"e6c349f1-c206-49d8-a9df-8ab81a3b68ea","Type":"ContainerStarted","Data":"f623f5de2fe5adf390656126671bf353d911dba48092be68ded917b97ab8cb2d"}
Jul 17 21:18:17.863678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.863100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zsxq7" event={"ID":"e6c349f1-c206-49d8-a9df-8ab81a3b68ea","Type":"ContainerStarted","Data":"6eb918c469b8fb3f88353c94a13c2626a910750c88eca01d4b384d6e62821b04"}
Jul 17 21:18:17.863678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.863619    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:18:17.866323 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.866235    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-rxq8w" event={"ID":"323ca98f-d27a-455c-9dea-22cfd7e88e9e","Type":"ContainerStarted","Data":"7a22b8b414ee1dabaabdf7e8319dcfa6f025b530b724a57ee7d054b59076aec6"}
Jul 17 21:18:17.869761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.869248    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qrm7f" event={"ID":"ec5ca31a-ad67-4400-88c4-829755a544dd","Type":"ContainerStarted","Data":"102252d2a5741f5287ab9ed03a98f990d4be772fbb3fc418eb6b7e3e5c1ce17f"}
Jul 17 21:18:17.869761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.869700    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:18:17.879550 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.879459    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-mwjff" podStartSLOduration=66.960207734 podStartE2EDuration="1m13.8794402s" totalImagesPullingTime="6.919232466s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:17.878810756 +0000 UTC m=+73.859181234" watchObservedRunningTime="2026-07-17 21:18:17.8794402 +0000 UTC m=+73.859810675"
Jul 17 21:18:17.897372 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.897318    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-zsxq7" podStartSLOduration=33.868880396 podStartE2EDuration="39.897302512s" totalImagesPullingTime="6.028422116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:17.896836899 +0000 UTC m=+73.877207372" watchObservedRunningTime="2026-07-17 21:18:17.897302512 +0000 UTC m=+73.877672967"
Jul 17 21:18:17.916622 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.916213    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-rxq8w" podStartSLOduration=17.782348182 podStartE2EDuration="33.916190322s" totalImagesPullingTime="16.13384214s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 21:17:44 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:17.915138994 +0000 UTC m=+73.895509472" watchObservedRunningTime="2026-07-17 21:18:17.916190322 +0000 UTC m=+73.896560799"
Jul 17 21:18:17.935788 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:17.935710    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qrm7f" podStartSLOduration=67.377670159 podStartE2EDuration="1m13.935690739s" totalImagesPullingTime="6.55802058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:17.933797157 +0000 UTC m=+73.914167633" watchObservedRunningTime="2026-07-17 21:18:17.935690739 +0000 UTC m=+73.916061265"
Jul 17 21:18:19.879074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:19.879026    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw" event={"ID":"dc59b820-a70f-43fb-81ee-d818b67c5c38","Type":"ContainerStarted","Data":"1c0e448686c9c9598590116a30cf8ed9fd7be951617407822117a093435a54ee"}
Jul 17 21:18:19.896856 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:19.896670    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-897466b88-5xmfw" podStartSLOduration=1.812831248 podStartE2EDuration="10.896646961s" totalImagesPullingTime="9.083815713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:19.896171365 +0000 UTC m=+75.876541844" watchObservedRunningTime="2026-07-17 21:18:19.896646961 +0000 UTC m=+75.877017439"
Jul 17 21:18:22.892126 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:22.892085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2" event={"ID":"52717cb2-c3ef-4d59-85ce-a7867fc8ba24","Type":"ContainerStarted","Data":"b07fda24910378becb240fcb5da6263336b79865e2c7895687d1e046fc1d020c"}
Jul 17 21:18:22.892576 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:22.892251    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:22.894157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:22.894135    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2"
Jul 17 21:18:22.917579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:22.917513    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-58c899dfc9-mjgv2" podStartSLOduration=2.125159107 podStartE2EDuration="13.917495982s" totalImagesPullingTime="11.792336875s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:22.916401927 +0000 UTC m=+78.896772415" watchObservedRunningTime="2026-07-17 21:18:22.917495982 +0000 UTC m=+78.897866458"
Jul 17 21:18:26.388460 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.388419    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-kdr5c"]
Jul 17 21:18:26.392098 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.392074    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.394788 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.394767    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 17 21:18:26.394997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.394973    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 17 21:18:26.395079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.395068    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p8ktt\""
Jul 17 21:18:26.395541 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.395523    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 17 21:18:26.395600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.395539    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 17 21:18:26.395821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.395795    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 17 21:18:26.395924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.395884    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 17 21:18:26.462145 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462114    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-tls\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462335 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462161    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-metrics-client-ca\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462335 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462187    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-root\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462335 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462210    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-textfile\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462335 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462265    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-wtmp\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462335 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462303    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462339    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lthx2\" (UniqueName: \"kubernetes.io/projected/a9a43db9-c576-4bd4-a25d-8af7ad92303c-kube-api-access-lthx2\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462361    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-sys\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.462596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.462382    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563242    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563310    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lthx2\" (UniqueName: \"kubernetes.io/projected/a9a43db9-c576-4bd4-a25d-8af7ad92303c-kube-api-access-lthx2\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-sys\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563369    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563441    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-tls\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563493    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-metrics-client-ca\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563517    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-root\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563541    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-textfile\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563576    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-wtmp\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563669    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-root\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563775    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-wtmp\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.563872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563829    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a9a43db9-c576-4bd4-a25d-8af7ad92303c-sys\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.564024 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563967    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-accelerators-collector-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.564024 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.563996    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-textfile\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.564219 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.564198    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a9a43db9-c576-4bd4-a25d-8af7ad92303c-metrics-client-ca\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.566085 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.566056    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-tls\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.566265 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.566248    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a9a43db9-c576-4bd4-a25d-8af7ad92303c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.574618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.574594    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lthx2\" (UniqueName: \"kubernetes.io/projected/a9a43db9-c576-4bd4-a25d-8af7ad92303c-kube-api-access-lthx2\") pod \"node-exporter-kdr5c\" (UID: \"a9a43db9-c576-4bd4-a25d-8af7ad92303c\") " pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.702579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.702479    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-kdr5c"
Jul 17 21:18:26.713811 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:26.713786    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a43db9_c576_4bd4_a25d_8af7ad92303c.slice/crio-e0a73fca71013f910721dc280398efde5f78cf5d673119f5d4f4e3ca5e141ff9 WatchSource:0}: Error finding container e0a73fca71013f910721dc280398efde5f78cf5d673119f5d4f4e3ca5e141ff9: Status 404 returned error can't find the container with id e0a73fca71013f910721dc280398efde5f78cf5d673119f5d4f4e3ca5e141ff9
Jul 17 21:18:26.907564 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:26.907530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdr5c" event={"ID":"a9a43db9-c576-4bd4-a25d-8af7ad92303c","Type":"ContainerStarted","Data":"e0a73fca71013f910721dc280398efde5f78cf5d673119f5d4f4e3ca5e141ff9"}
Jul 17 21:18:27.913045 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:27.912960    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdr5c" event={"ID":"a9a43db9-c576-4bd4-a25d-8af7ad92303c","Type":"ContainerStarted","Data":"4a21bcf4605d011fb1c19bbe0841e572ea9a1187fcaa28fb1ff5510d20109427"}
Jul 17 21:18:28.881599 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:28.881570    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zsxq7"
Jul 17 21:18:28.917656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:28.917620    2546 generic.go:356] "Generic (PLEG): container finished" podID="a9a43db9-c576-4bd4-a25d-8af7ad92303c" containerID="4a21bcf4605d011fb1c19bbe0841e572ea9a1187fcaa28fb1ff5510d20109427" exitCode=0
Jul 17 21:18:28.917656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:28.917661    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdr5c" event={"ID":"a9a43db9-c576-4bd4-a25d-8af7ad92303c","Type":"ContainerDied","Data":"4a21bcf4605d011fb1c19bbe0841e572ea9a1187fcaa28fb1ff5510d20109427"}
Jul 17 21:18:29.857484 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:29.857440    2546 patch_prober.go:28] interesting pod/image-registry-549479f6c5-v5cqb container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 17 21:18:29.857653 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:29.857498    2546 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb" podUID="b7d800b3-023c-498b-af08-9ec4308ea109" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:18:29.923501 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:29.923466    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdr5c" event={"ID":"a9a43db9-c576-4bd4-a25d-8af7ad92303c","Type":"ContainerStarted","Data":"d20e27a3e6eb77874c1e8b688de2db5195df1228611e93ab29b094f7d8dfa760"}
Jul 17 21:18:29.923924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:29.923551    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-kdr5c" event={"ID":"a9a43db9-c576-4bd4-a25d-8af7ad92303c","Type":"ContainerStarted","Data":"9861657c477b97b97dfcc093bbe948d388756194e03fd0d78d1e9214a5ec9170"}
Jul 17 21:18:29.959059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:29.959002    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-kdr5c" podStartSLOduration=2.807122185 podStartE2EDuration="3.958985558s" totalImagesPullingTime="1.079991563s" totalInitContainerRuntime="71.87181ms" isStatefulPod=true podCreationTimestamp="2026-07-17 21:18:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:29.956522151 +0000 UTC m=+85.936892650" watchObservedRunningTime="2026-07-17 21:18:29.958985558 +0000 UTC m=+85.939356034"
Jul 17 21:18:31.832675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:31.832646    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-549479f6c5-v5cqb"
Jul 17 21:18:34.049942 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.048169    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:18:34.052333 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.052309    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.055341 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.055320    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 17 21:18:34.055461 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.055381    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 17 21:18:34.056430 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056408    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 17 21:18:34.056654 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056415    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 17 21:18:34.056779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056422    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-pxj2w\""
Jul 17 21:18:34.056779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056482    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 17 21:18:34.056779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056484    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 17 21:18:34.056779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.056483    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 17 21:18:34.060859 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.060838    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 17 21:18:34.064612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.064591    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:18:34.120220 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120173    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120393 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120236    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120393 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120260    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120393 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120310    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120393 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120356    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120393 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.120599 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.120413    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52fsp\" (UniqueName: \"kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.220943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.220908    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.220952    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.220971    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.220989    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221028    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221053    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221075    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52fsp\" (UniqueName: \"kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221870 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221836    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.221985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221826    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.222028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221898    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.222028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.221987    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.223616 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.223586    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.223768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.223751    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.230625 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.230605    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-52fsp\" (UniqueName: \"kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp\") pod \"console-6c884c55cd-rcvls\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") " pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.362378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.362278    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:34.497282 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.497254    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:18:34.500524 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:18:34.500495    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81632a12_f5d4_4ad5_890e_e3dec1197e85.slice/crio-4eb1881780609ab1dd6cde19a5607c741712f8525d324f9010586d5d90f084e5 WatchSource:0}: Error finding container 4eb1881780609ab1dd6cde19a5607c741712f8525d324f9010586d5d90f084e5: Status 404 returned error can't find the container with id 4eb1881780609ab1dd6cde19a5607c741712f8525d324f9010586d5d90f084e5
Jul 17 21:18:34.941097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:34.941060    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c884c55cd-rcvls" event={"ID":"81632a12-f5d4-4ad5-890e-e3dec1197e85","Type":"ContainerStarted","Data":"4eb1881780609ab1dd6cde19a5607c741712f8525d324f9010586d5d90f084e5"}
Jul 17 21:18:37.957667 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:37.957630    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c884c55cd-rcvls" event={"ID":"81632a12-f5d4-4ad5-890e-e3dec1197e85","Type":"ContainerStarted","Data":"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"}
Jul 17 21:18:37.998466 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:37.998385    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c884c55cd-rcvls" podStartSLOduration=1.242298492 podStartE2EDuration="3.998366664s" totalImagesPullingTime="2.756068172s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:18:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:18:37.997203448 +0000 UTC m=+93.977573926" watchObservedRunningTime="2026-07-17 21:18:37.998366664 +0000 UTC m=+93.978737141"
Jul 17 21:18:44.362466 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:44.362426    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:44.363080 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:44.362526    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:44.367457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:44.367432    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:44.985315 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:44.985290    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:18:49.881597 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:18:49.881566    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qrm7f"
Jul 17 21:19:54.507208 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.507175    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:19:54.509302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.509286    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.537141 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.537116    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:19:54.587846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587813    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.587846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.587995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587860    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.587995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587878    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.587995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587929    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.587995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.587984    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zxvqt\" (UniqueName: \"kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.588118 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.588005    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689108    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689153    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zxvqt\" (UniqueName: \"kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689173    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689207    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689231    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689258    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689470 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689446    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.689944 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.689919    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.690088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.690057    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.690367 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.690170    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.690569 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.690542    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.692006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.691982    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.692098 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.691993    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.698337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.698314    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zxvqt\" (UniqueName: \"kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt\") pod \"console-85c48c5f5-lhbwq\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") " pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.817271 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.817220    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:19:54.953641 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:54.953613    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:19:54.957014 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:19:54.956986    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5040200e_5fbc_44e8_810b_9396ba9fbef0.slice/crio-6bb6b65207b457db4cb9a22edc7d0442bfd11ed1c54d878a859488dcebef04c2 WatchSource:0}: Error finding container 6bb6b65207b457db4cb9a22edc7d0442bfd11ed1c54d878a859488dcebef04c2: Status 404 returned error can't find the container with id 6bb6b65207b457db4cb9a22edc7d0442bfd11ed1c54d878a859488dcebef04c2
Jul 17 21:19:55.212760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:55.212654    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c48c5f5-lhbwq" event={"ID":"5040200e-5fbc-44e8-810b-9396ba9fbef0","Type":"ContainerStarted","Data":"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"}
Jul 17 21:19:55.212760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:55.212714    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c48c5f5-lhbwq" event={"ID":"5040200e-5fbc-44e8-810b-9396ba9fbef0","Type":"ContainerStarted","Data":"6bb6b65207b457db4cb9a22edc7d0442bfd11ed1c54d878a859488dcebef04c2"}
Jul 17 21:19:55.427797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:19:55.427716    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-85c48c5f5-lhbwq" podStartSLOduration=1.427699912 podStartE2EDuration="1.427699912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:19:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:19:55.425979049 +0000 UTC m=+171.406349528" watchObservedRunningTime="2026-07-17 21:19:55.427699912 +0000 UTC m=+171.408070459"
Jul 17 21:20:04.817650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:04.817614    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:20:04.818104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:04.817682    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:20:04.822403 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:04.822378    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:20:05.252789 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:05.252764    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:20:05.315231 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:05.315203    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:20:30.336314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.336243    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c884c55cd-rcvls" podUID="81632a12-f5d4-4ad5-890e-e3dec1197e85" containerName="console" containerID="cri-o://b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64" gracePeriod=15
Jul 17 21:20:30.577097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.577075    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:20:30.733308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733234    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733281    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733305    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733442    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733512    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733550    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-52fsp\" (UniqueName: \"kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733704 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733581    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert\") pod \"81632a12-f5d4-4ad5-890e-e3dec1197e85\" (UID: \"81632a12-f5d4-4ad5-890e-e3dec1197e85\") "
Jul 17 21:20:30.733783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733628    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config" (OuterVolumeSpecName: "console-config") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:20:30.733783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733763    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:20:30.733878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733840    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.733878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733857    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-trusted-ca-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.733964 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.733945    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca" (OuterVolumeSpecName: "service-ca") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:20:30.734085 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.734062    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:20:30.735651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.735630    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:20:30.735929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.735906    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp" (OuterVolumeSpecName: "kube-api-access-52fsp") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "kube-api-access-52fsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:20:30.735988 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.735959    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "81632a12-f5d4-4ad5-890e-e3dec1197e85" (UID: "81632a12-f5d4-4ad5-890e-e3dec1197e85"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:20:30.834906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.834882    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.834906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.834907    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.835058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.834916    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/81632a12-f5d4-4ad5-890e-e3dec1197e85-console-oauth-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.835058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.834925    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-52fsp\" (UniqueName: \"kubernetes.io/projected/81632a12-f5d4-4ad5-890e-e3dec1197e85-kube-api-access-52fsp\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:30.835058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:30.834934    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/81632a12-f5d4-4ad5-890e-e3dec1197e85-oauth-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:20:31.344394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.344356    2546 generic.go:356] "Generic (PLEG): container finished" podID="81632a12-f5d4-4ad5-890e-e3dec1197e85" containerID="b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64" exitCode=0
Jul 17 21:20:31.344394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.344399    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c884c55cd-rcvls"
Jul 17 21:20:31.344886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.344439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c884c55cd-rcvls" event={"ID":"81632a12-f5d4-4ad5-890e-e3dec1197e85","Type":"ContainerDied","Data":"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"}
Jul 17 21:20:31.344886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.344491    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c884c55cd-rcvls" event={"ID":"81632a12-f5d4-4ad5-890e-e3dec1197e85","Type":"ContainerDied","Data":"4eb1881780609ab1dd6cde19a5607c741712f8525d324f9010586d5d90f084e5"}
Jul 17 21:20:31.344886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.344513    2546 scope.go:117] "RemoveContainer" containerID="b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"
Jul 17 21:20:31.353697 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.353677    2546 scope.go:117] "RemoveContainer" containerID="b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"
Jul 17 21:20:31.353968 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:20:31.353949    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64\": container with ID starting with b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64 not found: ID does not exist" containerID="b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"
Jul 17 21:20:31.354026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.353976    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64"} err="failed to get container status \"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64\": rpc error: code = NotFound desc = could not find container \"b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64\": container with ID starting with b9d2f4731ee835ab37d8dbb2cda6f9a992e2d467f5339e564d49824936bb0d64 not found: ID does not exist"
Jul 17 21:20:31.383886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.383864    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:20:31.388740 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:31.388697    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c884c55cd-rcvls"]
Jul 17 21:20:32.535231 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:20:32.535200    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81632a12-f5d4-4ad5-890e-e3dec1197e85" path="/var/lib/kubelet/pods/81632a12-f5d4-4ad5-890e-e3dec1197e85/volumes"
Jul 17 21:21:15.907304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.907274    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:21:15.907800 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.907501    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81632a12-f5d4-4ad5-890e-e3dec1197e85" containerName="console"
Jul 17 21:21:15.907800 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.907511    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="81632a12-f5d4-4ad5-890e-e3dec1197e85" containerName="console"
Jul 17 21:21:15.907800 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.907562    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="81632a12-f5d4-4ad5-890e-e3dec1197e85" containerName="console"
Jul 17 21:21:15.909968 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.909949    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.936761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.936735    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.936903 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.936850    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.936903 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.936893    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.937001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.936929    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.937001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.936965    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.937087 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.937069    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k2lx\" (UniqueName: \"kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.937131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.937094    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:15.939026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:15.939007    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:21:16.038113 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038090    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k2lx\" (UniqueName: \"kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038212 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038184    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038272 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038230    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038272 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038267    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038282    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038302    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038332    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.038879 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038846    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.039004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.038981    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.039108 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.039090    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.039222 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.039102    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.040807 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.040782    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.040886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.040862    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.053415 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.053397    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k2lx\" (UniqueName: \"kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx\") pod \"console-9b79c56dc-rswbn\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") " pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.219195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.219142    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:16.358433 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.358404    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:21:16.361626 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:21:16.361598    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0811e21b_2d9e_4f7e_b588_29561e7ace39.slice/crio-89e7d7dfc919089255fe1381892dc438a0370e0d563a7fa1ded477665da7ac24 WatchSource:0}: Error finding container 89e7d7dfc919089255fe1381892dc438a0370e0d563a7fa1ded477665da7ac24: Status 404 returned error can't find the container with id 89e7d7dfc919089255fe1381892dc438a0370e0d563a7fa1ded477665da7ac24
Jul 17 21:21:16.504830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.504804    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b79c56dc-rswbn" event={"ID":"0811e21b-2d9e-4f7e-b588-29561e7ace39","Type":"ContainerStarted","Data":"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"}
Jul 17 21:21:16.504996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.504849    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b79c56dc-rswbn" event={"ID":"0811e21b-2d9e-4f7e-b588-29561e7ace39","Type":"ContainerStarted","Data":"89e7d7dfc919089255fe1381892dc438a0370e0d563a7fa1ded477665da7ac24"}
Jul 17 21:21:16.538392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:16.538347    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-9b79c56dc-rswbn" podStartSLOduration=1.538334122 podStartE2EDuration="1.538334122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:21:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:21:16.536371694 +0000 UTC m=+252.516742172" watchObservedRunningTime="2026-07-17 21:21:16.538334122 +0000 UTC m=+252.518704598"
Jul 17 21:21:26.219627 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:26.219598    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:26.220017 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:26.219647    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:26.223997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:26.223978    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:26.540008 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:26.539983    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:21:26.592684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:26.592659    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:21:51.614537 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.614497    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-85c48c5f5-lhbwq" podUID="5040200e-5fbc-44e8-810b-9396ba9fbef0" containerName="console" containerID="cri-o://d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156" gracePeriod=15
Jul 17 21:21:51.852353 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.852333    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:21:51.978482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978421    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978454    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zxvqt\" (UniqueName: \"kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978474    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978666 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978496    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978666 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978529    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978659    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978700    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle\") pod \"5040200e-5fbc-44e8-810b-9396ba9fbef0\" (UID: \"5040200e-5fbc-44e8-810b-9396ba9fbef0\") "
Jul 17 21:21:51.978946 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978918    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca" (OuterVolumeSpecName: "service-ca") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:21:51.979030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.978937    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config" (OuterVolumeSpecName: "console-config") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:21:51.979148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.979126    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:21:51.979205 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.979164    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:21:51.980655 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.980635    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:21:51.981066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.981038    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:21:51.981121 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:51.981066    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt" (OuterVolumeSpecName: "kube-api-access-zxvqt") pod "5040200e-5fbc-44e8-810b-9396ba9fbef0" (UID: "5040200e-5fbc-44e8-810b-9396ba9fbef0"). InnerVolumeSpecName "kube-api-access-zxvqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:21:52.079439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079416    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-oauth-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079436    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-trusted-ca-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079445    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079454    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zxvqt\" (UniqueName: \"kubernetes.io/projected/5040200e-5fbc-44e8-810b-9396ba9fbef0-kube-api-access-zxvqt\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079462    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079471    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.079557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.079478    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/5040200e-5fbc-44e8-810b-9396ba9fbef0-console-oauth-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:21:52.615577 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.615546    2546 generic.go:356] "Generic (PLEG): container finished" podID="5040200e-5fbc-44e8-810b-9396ba9fbef0" containerID="d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156" exitCode=0
Jul 17 21:21:52.616001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.615608    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-85c48c5f5-lhbwq"
Jul 17 21:21:52.616001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.615611    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c48c5f5-lhbwq" event={"ID":"5040200e-5fbc-44e8-810b-9396ba9fbef0","Type":"ContainerDied","Data":"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"}
Jul 17 21:21:52.616001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.615658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-85c48c5f5-lhbwq" event={"ID":"5040200e-5fbc-44e8-810b-9396ba9fbef0","Type":"ContainerDied","Data":"6bb6b65207b457db4cb9a22edc7d0442bfd11ed1c54d878a859488dcebef04c2"}
Jul 17 21:21:52.616001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.615685    2546 scope.go:117] "RemoveContainer" containerID="d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"
Jul 17 21:21:52.623072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.623058    2546 scope.go:117] "RemoveContainer" containerID="d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"
Jul 17 21:21:52.623284 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:21:52.623266    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156\": container with ID starting with d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156 not found: ID does not exist" containerID="d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"
Jul 17 21:21:52.623344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.623293    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156"} err="failed to get container status \"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156\": rpc error: code = NotFound desc = could not find container \"d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156\": container with ID starting with d0f19dcfe64faa2f3170223b09a0ccc3447afb24e580e7e0121a3fd718bb6156 not found: ID does not exist"
Jul 17 21:21:52.630849 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.630829    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:21:52.636594 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:52.636577    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-85c48c5f5-lhbwq"]
Jul 17 21:21:54.534531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:21:54.534489    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5040200e-5fbc-44e8-810b-9396ba9fbef0" path="/var/lib/kubelet/pods/5040200e-5fbc-44e8-810b-9396ba9fbef0/volumes"
Jul 17 21:22:04.402971 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:04.402945    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:22:04.407967 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:04.407948    2546 kubelet.go:1648] "Image garbage collection succeeded"
Jul 17 21:22:04.411300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:04.411281    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:22:11.363739 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.363692    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"]
Jul 17 21:22:11.366014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.363963    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5040200e-5fbc-44e8-810b-9396ba9fbef0" containerName="console"
Jul 17 21:22:11.366014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.363974    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5040200e-5fbc-44e8-810b-9396ba9fbef0" containerName="console"
Jul 17 21:22:11.366014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.364021    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5040200e-5fbc-44e8-810b-9396ba9fbef0" containerName="console"
Jul 17 21:22:11.366898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.366882    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.369602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.369581    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:22:11.369757 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.369694    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:22:11.370372 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.370356    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-94drt\""
Jul 17 21:22:11.375815 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.375795    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"]
Jul 17 21:22:11.508458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.508399    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.508458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.508466    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcxbx\" (UniqueName: \"kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.508678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.508574    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.609429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.609385    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcxbx\" (UniqueName: \"kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.609429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.609451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.609656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.609494    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.609936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.609912    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.609936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.609927    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.618417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.618361    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcxbx\" (UniqueName: \"kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.675862 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.675828    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:11.806742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.806687    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"]
Jul 17 21:22:11.810299 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:11.810268    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5343962b_6e36_48bb_81a0_fda1abb9114b.slice/crio-119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64 WatchSource:0}: Error finding container 119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64: Status 404 returned error can't find the container with id 119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64
Jul 17 21:22:11.812223 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:11.812205    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:22:12.683101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:12.683042    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerStarted","Data":"119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64"}
Jul 17 21:22:17.702651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:17.702617    2546 generic.go:356] "Generic (PLEG): container finished" podID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerID="6126f8ba3250fd2c39a685932d64c81795e18812714a46f64197c15861f3a502" exitCode=0
Jul 17 21:22:17.702970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:17.702697    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerDied","Data":"6126f8ba3250fd2c39a685932d64c81795e18812714a46f64197c15861f3a502"}
Jul 17 21:22:20.715127 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:20.715094    2546 generic.go:356] "Generic (PLEG): container finished" podID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerID="d889b3b64ea9971fb919e03eac72402c25f19718db2b4f04799e798c00710c47" exitCode=0
Jul 17 21:22:20.715582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:20.715186    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerDied","Data":"d889b3b64ea9971fb919e03eac72402c25f19718db2b4f04799e798c00710c47"}
Jul 17 21:22:27.743547 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:27.743516    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerStarted","Data":"8a65d6f3a5725ad97f5bb52ac765d88e42eb3771b7e48131596c7a3761bb6a68"}
Jul 17 21:22:27.762864 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:27.762818    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" podStartSLOduration=1.94748592 podStartE2EDuration="16.76280325s" totalImagesPullingTime="14.81531733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:22:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:22:27.761587682 +0000 UTC m=+323.741958158" watchObservedRunningTime="2026-07-17 21:22:27.76280325 +0000 UTC m=+323.743173744"
Jul 17 21:22:28.748409 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:28.748375    2546 generic.go:356] "Generic (PLEG): container finished" podID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerID="8a65d6f3a5725ad97f5bb52ac765d88e42eb3771b7e48131596c7a3761bb6a68" exitCode=0
Jul 17 21:22:28.748791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:28.748434    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerDied","Data":"8a65d6f3a5725ad97f5bb52ac765d88e42eb3771b7e48131596c7a3761bb6a68"}
Jul 17 21:22:29.878417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:29.878398    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:30.044623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.044536    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle\") pod \"5343962b-6e36-48bb-81a0-fda1abb9114b\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") "
Jul 17 21:22:30.044623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.044575    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util\") pod \"5343962b-6e36-48bb-81a0-fda1abb9114b\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") "
Jul 17 21:22:30.044623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.044603    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mcxbx\" (UniqueName: \"kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx\") pod \"5343962b-6e36-48bb-81a0-fda1abb9114b\" (UID: \"5343962b-6e36-48bb-81a0-fda1abb9114b\") "
Jul 17 21:22:30.045201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.045178    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle" (OuterVolumeSpecName: "bundle") pod "5343962b-6e36-48bb-81a0-fda1abb9114b" (UID: "5343962b-6e36-48bb-81a0-fda1abb9114b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:22:30.046948 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.046923    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx" (OuterVolumeSpecName: "kube-api-access-mcxbx") pod "5343962b-6e36-48bb-81a0-fda1abb9114b" (UID: "5343962b-6e36-48bb-81a0-fda1abb9114b"). InnerVolumeSpecName "kube-api-access-mcxbx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:22:30.049080 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.049047    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util" (OuterVolumeSpecName: "util") pod "5343962b-6e36-48bb-81a0-fda1abb9114b" (UID: "5343962b-6e36-48bb-81a0-fda1abb9114b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:22:30.145189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.145156    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:30.145189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.145183    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5343962b-6e36-48bb-81a0-fda1abb9114b-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:30.145189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.145192    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcxbx\" (UniqueName: \"kubernetes.io/projected/5343962b-6e36-48bb-81a0-fda1abb9114b-kube-api-access-mcxbx\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:30.756233 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.756193    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7" event={"ID":"5343962b-6e36-48bb-81a0-fda1abb9114b","Type":"ContainerDied","Data":"119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64"}
Jul 17 21:22:30.756387 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.756284    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="119cf343112ff7a3efa7568eac41c07a599d04834330f2de5447ae79701abb64"
Jul 17 21:22:30.756387 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:30.756208    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540r7pr7"
Jul 17 21:22:34.425862 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.425829    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"]
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426061    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="extract"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426071    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="extract"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426090    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="util"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426094    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="util"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426101    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="pull"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426106    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="pull"
Jul 17 21:22:34.426302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.426143    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5343962b-6e36-48bb-81a0-fda1abb9114b" containerName="extract"
Jul 17 21:22:34.453344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.453314    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"]
Jul 17 21:22:34.453503 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.453424    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.456558 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.456535    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-jsmf9\""
Jul 17 21:22:34.457510 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.457495    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 17 21:22:34.457754 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.457738    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:22:34.475285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.475258    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgz27\" (UniqueName: \"kubernetes.io/projected/d8771ae5-8d51-4d20-aca0-b844e9712ad2-kube-api-access-zgz27\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.475368 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.475299    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8771ae5-8d51-4d20-aca0-b844e9712ad2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.575864 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.575834    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgz27\" (UniqueName: \"kubernetes.io/projected/d8771ae5-8d51-4d20-aca0-b844e9712ad2-kube-api-access-zgz27\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.576039 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.575899    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8771ae5-8d51-4d20-aca0-b844e9712ad2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.576245 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.576228    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d8771ae5-8d51-4d20-aca0-b844e9712ad2-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.587276 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.587250    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgz27\" (UniqueName: \"kubernetes.io/projected/d8771ae5-8d51-4d20-aca0-b844e9712ad2-kube-api-access-zgz27\") pod \"cert-manager-operator-controller-manager-55c444bf86-p72nb\" (UID: \"d8771ae5-8d51-4d20-aca0-b844e9712ad2\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.762473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.762446    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"
Jul 17 21:22:34.915703 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:34.915679    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb"]
Jul 17 21:22:34.917976 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:34.917945    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8771ae5_8d51_4d20_aca0_b844e9712ad2.slice/crio-e1a73a4540f8f47214383437f4ceea216931ec320657275f9de73269b8f7b510 WatchSource:0}: Error finding container e1a73a4540f8f47214383437f4ceea216931ec320657275f9de73269b8f7b510: Status 404 returned error can't find the container with id e1a73a4540f8f47214383437f4ceea216931ec320657275f9de73269b8f7b510
Jul 17 21:22:35.774206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:35.774172    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb" event={"ID":"d8771ae5-8d51-4d20-aca0-b844e9712ad2","Type":"ContainerStarted","Data":"e1a73a4540f8f47214383437f4ceea216931ec320657275f9de73269b8f7b510"}
Jul 17 21:22:38.787474 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:38.787439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb" event={"ID":"d8771ae5-8d51-4d20-aca0-b844e9712ad2","Type":"ContainerStarted","Data":"f6575f18274b008474100c3fa1ca2393a5539272c8334bf51a9c03cd726594cb"}
Jul 17 21:22:38.808681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:38.808614    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-p72nb" podStartSLOduration=1.425098738 podStartE2EDuration="4.808586107s" totalImagesPullingTime="3.383487369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:22:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:22:38.807581317 +0000 UTC m=+334.787951793" watchObservedRunningTime="2026-07-17 21:22:38.808586107 +0000 UTC m=+334.788956584"
Jul 17 21:22:39.451061 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.451027    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"]
Jul 17 21:22:39.465859 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.465835    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.468447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.468427    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-94drt\""
Jul 17 21:22:39.469207 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.469191    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:22:39.469207 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.469202    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:22:39.471616 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.471582    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"]
Jul 17 21:22:39.504127 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.504097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.504237 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.504135    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.504237 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.504181    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jm6f\" (UniqueName: \"kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.604647 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.604613    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.604830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.604659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.604830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.604761    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jm6f\" (UniqueName: \"kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.605028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.605011    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.605079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.605062    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.627527 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.627493    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jm6f\" (UniqueName: \"kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.775777 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.775736    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:39.929120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:39.929096    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"]
Jul 17 21:22:39.931168 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:39.931142    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731da1e1_aa8f_4a4b_b8ea_6210655a6b20.slice/crio-64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185 WatchSource:0}: Error finding container 64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185: Status 404 returned error can't find the container with id 64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185
Jul 17 21:22:40.795390 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:40.795356    2546 generic.go:356] "Generic (PLEG): container finished" podID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerID="ea65daa9ddc5a8fbcd9fb41ff0f2e3290afb540bdb77ef5b583e94723329d133" exitCode=0
Jul 17 21:22:40.795570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:40.795448    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerDied","Data":"ea65daa9ddc5a8fbcd9fb41ff0f2e3290afb540bdb77ef5b583e94723329d133"}
Jul 17 21:22:40.795570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:40.795507    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerStarted","Data":"64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185"}
Jul 17 21:22:41.872036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.872002    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"]
Jul 17 21:22:41.874180 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.874152    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:41.876591 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.876566    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 17 21:22:41.877362 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.877339    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-p6w5h\""
Jul 17 21:22:41.877445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.877340    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 17 21:22:41.883545 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.883464    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"]
Jul 17 21:22:41.921556 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.921514    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrtlj\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-kube-api-access-zrtlj\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:41.921762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:41.921618    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.022404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.022363    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrtlj\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-kube-api-access-zrtlj\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.022581 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.022425    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.032586 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.032555    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrtlj\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-kube-api-access-zrtlj\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.034341 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.034312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/44d7662f-83d1-43c1-b302-b00062f4d2ee-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-7qpn8\" (UID: \"44d7662f-83d1-43c1-b302-b00062f4d2ee\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.185227 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.185147    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"
Jul 17 21:22:42.807716 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.807685    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerStarted","Data":"34de7b61ea83719d3cdb8e91c939731d47241b95594ef78112fe562aa975c04c"}
Jul 17 21:22:42.808021 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:42.807998    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8"]
Jul 17 21:22:42.811368 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:42.811342    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod44d7662f_83d1_43c1_b302_b00062f4d2ee.slice/crio-53843a48ada80cc72abedbcd8ffa3d1dd0c23de9367f8135252c2fb158eb4f18 WatchSource:0}: Error finding container 53843a48ada80cc72abedbcd8ffa3d1dd0c23de9367f8135252c2fb158eb4f18: Status 404 returned error can't find the container with id 53843a48ada80cc72abedbcd8ffa3d1dd0c23de9367f8135252c2fb158eb4f18
Jul 17 21:22:43.816963 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:43.816922    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8" event={"ID":"44d7662f-83d1-43c1-b302-b00062f4d2ee","Type":"ContainerStarted","Data":"53843a48ada80cc72abedbcd8ffa3d1dd0c23de9367f8135252c2fb158eb4f18"}
Jul 17 21:22:43.818869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:43.818837    2546 generic.go:356] "Generic (PLEG): container finished" podID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerID="34de7b61ea83719d3cdb8e91c939731d47241b95594ef78112fe562aa975c04c" exitCode=0
Jul 17 21:22:43.818998 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:43.818877    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerDied","Data":"34de7b61ea83719d3cdb8e91c939731d47241b95594ef78112fe562aa975c04c"}
Jul 17 21:22:43.977164 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:22:43.977118    2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731da1e1_aa8f_4a4b_b8ea_6210655a6b20.slice/crio-8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731da1e1_aa8f_4a4b_b8ea_6210655a6b20.slice/crio-conmon-8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 21:22:43.977322 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:22:43.977123    2546 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731da1e1_aa8f_4a4b_b8ea_6210655a6b20.slice/crio-8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod731da1e1_aa8f_4a4b_b8ea_6210655a6b20.slice/crio-conmon-8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844.scope\": RecentStats: unable to find data in memory cache]"
Jul 17 21:22:44.826012 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:44.825968    2546 generic.go:356] "Generic (PLEG): container finished" podID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerID="8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844" exitCode=0
Jul 17 21:22:44.826482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:44.826018    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerDied","Data":"8617e858165d50d88942debb7bc332a35fc2918256c46d6a8c60b97881006844"}
Jul 17 21:22:45.830772 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:45.830732    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8" event={"ID":"44d7662f-83d1-43c1-b302-b00062f4d2ee","Type":"ContainerStarted","Data":"c8894d2423b00ca326e6fa523a0cae159719e303785af95e270f22816e124b69"}
Jul 17 21:22:45.850102 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:45.850046    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-7qpn8" podStartSLOduration=2.302312223 podStartE2EDuration="4.850026047s" totalImagesPullingTime="2.547713824s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:22:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:22:45.847426692 +0000 UTC m=+341.827797168" watchObservedRunningTime="2026-07-17 21:22:45.850026047 +0000 UTC m=+341.830396525"
Jul 17 21:22:45.955685 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:45.955660    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:46.049930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.049909    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle\") pod \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") "
Jul 17 21:22:46.050057 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.049950    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6jm6f\" (UniqueName: \"kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f\") pod \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") "
Jul 17 21:22:46.050057 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.049996    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util\") pod \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\" (UID: \"731da1e1-aa8f-4a4b-b8ea-6210655a6b20\") "
Jul 17 21:22:46.050346 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.050320    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle" (OuterVolumeSpecName: "bundle") pod "731da1e1-aa8f-4a4b-b8ea-6210655a6b20" (UID: "731da1e1-aa8f-4a4b-b8ea-6210655a6b20"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:22:46.052083 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.052062    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f" (OuterVolumeSpecName: "kube-api-access-6jm6f") pod "731da1e1-aa8f-4a4b-b8ea-6210655a6b20" (UID: "731da1e1-aa8f-4a4b-b8ea-6210655a6b20"). InnerVolumeSpecName "kube-api-access-6jm6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:22:46.054619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.054598    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util" (OuterVolumeSpecName: "util") pod "731da1e1-aa8f-4a4b-b8ea-6210655a6b20" (UID: "731da1e1-aa8f-4a4b-b8ea-6210655a6b20"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:22:46.150517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.150467    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:46.150517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.150486    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jm6f\" (UniqueName: \"kubernetes.io/projected/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-kube-api-access-6jm6f\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:46.150517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.150495    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/731da1e1-aa8f-4a4b-b8ea-6210655a6b20-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:22:46.705444 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705413    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z7bxw"]
Jul 17 21:22:46.705670 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705655    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="extract"
Jul 17 21:22:46.705711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705675    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="extract"
Jul 17 21:22:46.705711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705685    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="pull"
Jul 17 21:22:46.705711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705690    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="pull"
Jul 17 21:22:46.705711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705699    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="util"
Jul 17 21:22:46.705711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705704    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="util"
Jul 17 21:22:46.705886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.705775    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="731da1e1-aa8f-4a4b-b8ea-6210655a6b20" containerName="extract"
Jul 17 21:22:46.707669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.707653    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.711457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.711436    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-mxl4k\""
Jul 17 21:22:46.723794 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.723777    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z7bxw"]
Jul 17 21:22:46.754637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.754610    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.754788 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.754673    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jb76h\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-kube-api-access-jb76h\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.836897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.836845    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm"
Jul 17 21:22:46.836897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.836867    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fzm9qm" event={"ID":"731da1e1-aa8f-4a4b-b8ea-6210655a6b20","Type":"ContainerDied","Data":"64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185"}
Jul 17 21:22:46.837218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.836906    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64249ca646b4d994c602b8d11db5a7dab804e235cb6e26582cae0a8ea2662185"
Jul 17 21:22:46.860474 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.860450    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.860568 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.860504    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jb76h\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-kube-api-access-jb76h\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.869058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.869033    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jb76h\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-kube-api-access-jb76h\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:46.869136 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:46.869056    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/567a03c1-eeb6-4b22-9d49-36e2346524f0-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-z7bxw\" (UID: \"567a03c1-eeb6-4b22-9d49-36e2346524f0\") " pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:47.015581 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.015553    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:47.163308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.163284    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-z7bxw"]
Jul 17 21:22:47.165483 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:47.165455    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod567a03c1_eeb6_4b22_9d49_36e2346524f0.slice/crio-ce9ba7568bfccd2e81450fc6e9106b784c601d10c746c1be90a0a188d3edda6d WatchSource:0}: Error finding container ce9ba7568bfccd2e81450fc6e9106b784c601d10c746c1be90a0a188d3edda6d: Status 404 returned error can't find the container with id ce9ba7568bfccd2e81450fc6e9106b784c601d10c746c1be90a0a188d3edda6d
Jul 17 21:22:47.842591 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.842555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw" event={"ID":"567a03c1-eeb6-4b22-9d49-36e2346524f0","Type":"ContainerStarted","Data":"f1a40db27f3c42939e707e73de73dc4a125326618a027515045948217971be8a"}
Jul 17 21:22:47.843011 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.842603    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw" event={"ID":"567a03c1-eeb6-4b22-9d49-36e2346524f0","Type":"ContainerStarted","Data":"ce9ba7568bfccd2e81450fc6e9106b784c601d10c746c1be90a0a188d3edda6d"}
Jul 17 21:22:47.843011 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.842635    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:47.859684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:47.859638    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw" podStartSLOduration=1.859624846 podStartE2EDuration="1.859624846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:22:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:22:47.858851894 +0000 UTC m=+343.839222364" watchObservedRunningTime="2026-07-17 21:22:47.859624846 +0000 UTC m=+343.839995323"
Jul 17 21:22:53.848916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:53.848887    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-z7bxw"
Jul 17 21:22:58.609487 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.609414    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"]
Jul 17 21:22:58.611790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.611773    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.614282 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.614262    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 17 21:22:58.615812 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.615793    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-94drt\""
Jul 17 21:22:58.615913 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.615814    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 17 21:22:58.617596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.617573    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"]
Jul 17 21:22:58.639875 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.639852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drfhz\" (UniqueName: \"kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.639970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.639888    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.639970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.639919    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.741189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.741165    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drfhz\" (UniqueName: \"kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.741344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.741209    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.741344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.741258    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.741574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.741555    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.741632 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.741588    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.749738 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.749691    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drfhz\" (UniqueName: \"kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:58.922250 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:58.922175    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:22:59.047103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:59.047073    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"]
Jul 17 21:22:59.049960 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:22:59.049935    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6759db_dbaf_446e_8dd3_b348b4caf4ae.slice/crio-3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf WatchSource:0}: Error finding container 3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf: Status 404 returned error can't find the container with id 3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf
Jul 17 21:22:59.892336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:59.892299    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerID="295cdcf71f208d076b92b72251ca24410367e519b67942c1be4e0d67062082e5" exitCode=0
Jul 17 21:22:59.892695 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:59.892358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq" event={"ID":"4d6759db-dbaf-446e-8dd3-b348b4caf4ae","Type":"ContainerDied","Data":"295cdcf71f208d076b92b72251ca24410367e519b67942c1be4e0d67062082e5"}
Jul 17 21:22:59.892695 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:22:59.892436    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq" event={"ID":"4d6759db-dbaf-446e-8dd3-b348b4caf4ae","Type":"ContainerStarted","Data":"3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf"}
Jul 17 21:23:00.898388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:00.898305    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerID="158ddbec5b986f7578017f2d4996c0d50c89d15d127798e0de8e4b4de2bb6dd4" exitCode=0
Jul 17 21:23:00.898388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:00.898361    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq" event={"ID":"4d6759db-dbaf-446e-8dd3-b348b4caf4ae","Type":"ContainerDied","Data":"158ddbec5b986f7578017f2d4996c0d50c89d15d127798e0de8e4b4de2bb6dd4"}
Jul 17 21:23:01.905153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:01.905119    2546 generic.go:356] "Generic (PLEG): container finished" podID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerID="cb20f6dd6e16ffa4a0ce06c40271b839d6a93ae7e1d40971f0721c8955681b99" exitCode=0
Jul 17 21:23:01.905530 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:01.905213    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq" event={"ID":"4d6759db-dbaf-446e-8dd3-b348b4caf4ae","Type":"ContainerDied","Data":"cb20f6dd6e16ffa4a0ce06c40271b839d6a93ae7e1d40971f0721c8955681b99"}
Jul 17 21:23:03.026803 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.026781    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:23:03.074897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.074865    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle\") pod \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") "
Jul 17 21:23:03.075042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.074936    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drfhz\" (UniqueName: \"kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz\") pod \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") "
Jul 17 21:23:03.075042 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.074969    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util\") pod \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\" (UID: \"4d6759db-dbaf-446e-8dd3-b348b4caf4ae\") "
Jul 17 21:23:03.075654 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.075624    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle" (OuterVolumeSpecName: "bundle") pod "4d6759db-dbaf-446e-8dd3-b348b4caf4ae" (UID: "4d6759db-dbaf-446e-8dd3-b348b4caf4ae"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:03.077037 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.077019    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz" (OuterVolumeSpecName: "kube-api-access-drfhz") pod "4d6759db-dbaf-446e-8dd3-b348b4caf4ae" (UID: "4d6759db-dbaf-446e-8dd3-b348b4caf4ae"). InnerVolumeSpecName "kube-api-access-drfhz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:23:03.080834 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.080812    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util" (OuterVolumeSpecName: "util") pod "4d6759db-dbaf-446e-8dd3-b348b4caf4ae" (UID: "4d6759db-dbaf-446e-8dd3-b348b4caf4ae"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:03.175505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.175445    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:03.175505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.175469    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drfhz\" (UniqueName: \"kubernetes.io/projected/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-kube-api-access-drfhz\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:03.175505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.175481    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/4d6759db-dbaf-446e-8dd3-b348b4caf4ae-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:03.915056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.915024    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq" event={"ID":"4d6759db-dbaf-446e-8dd3-b348b4caf4ae","Type":"ContainerDied","Data":"3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf"}
Jul 17 21:23:03.915217 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.915079    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e344b9b5b3d4aeb74572827defb8a976b8d8ad2e3b10bae62fc2cc47505c2bf"
Jul 17 21:23:03.915217 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:03.915034    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796aflmmqq"
Jul 17 21:23:15.827820 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.827781    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"]
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828082    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="util"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828094    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="util"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828106    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="pull"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828111    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="pull"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828117    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="extract"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828147    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="extract"
Jul 17 21:23:15.828236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.828214    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6759db-dbaf-446e-8dd3-b348b4caf4ae" containerName="extract"
Jul 17 21:23:15.831737 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.831709    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.835018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.834998    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-x6d45\""
Jul 17 21:23:15.835237 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.835224    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 17 21:23:15.835336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.835318    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 17 21:23:15.835908 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.835893    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 17 21:23:15.836675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.836658    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 17 21:23:15.863123 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.863100    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"]
Jul 17 21:23:15.866949 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.866926    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-manifests\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.867060 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.866954    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-tmp\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.867060 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.866988    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.867060 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.867044    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.867224 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.867080    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tk9sh\" (UniqueName: \"kubernetes.io/projected/789287cc-65cb-4d08-9c36-320a8146443e-kube-api-access-tk9sh\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.967994 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.967958    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.967994 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.968006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.968209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.968140    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tk9sh\" (UniqueName: \"kubernetes.io/projected/789287cc-65cb-4d08-9c36-320a8146443e-kube-api-access-tk9sh\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.968209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.968201    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-manifests\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.968284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.968226    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-tmp\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.968548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.968530    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-manifests\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.970454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.970435    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/789287cc-65cb-4d08-9c36-320a8146443e-tmp\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.970651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.970627    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.970762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.970631    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/789287cc-65cb-4d08-9c36-320a8146443e-webhook-cert\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:15.989787 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:15.989766    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tk9sh\" (UniqueName: \"kubernetes.io/projected/789287cc-65cb-4d08-9c36-320a8146443e-kube-api-access-tk9sh\") pod \"opendatahub-operator-controller-manager-6d6dbd9688-wf99g\" (UID: \"789287cc-65cb-4d08-9c36-320a8146443e\") " pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:16.141540 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:16.141456    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:16.281423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:16.281400    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"]
Jul 17 21:23:16.284100 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:23:16.284074    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod789287cc_65cb_4d08_9c36_320a8146443e.slice/crio-109cb18e46353c9c31427c63950ea6bb32de85a2749e87b0a90d2f546760f396 WatchSource:0}: Error finding container 109cb18e46353c9c31427c63950ea6bb32de85a2749e87b0a90d2f546760f396: Status 404 returned error can't find the container with id 109cb18e46353c9c31427c63950ea6bb32de85a2749e87b0a90d2f546760f396
Jul 17 21:23:16.972394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:16.972311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g" event={"ID":"789287cc-65cb-4d08-9c36-320a8146443e","Type":"ContainerStarted","Data":"109cb18e46353c9c31427c63950ea6bb32de85a2749e87b0a90d2f546760f396"}
Jul 17 21:23:19.987120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:19.987083    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g" event={"ID":"789287cc-65cb-4d08-9c36-320a8146443e","Type":"ContainerStarted","Data":"147e05257ba9a749561ca145c7600b33687653870b06942d45cac5e1c08917b5"}
Jul 17 21:23:20.993090 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:20.993055    2546 generic.go:356] "Generic (PLEG): container finished" podID="789287cc-65cb-4d08-9c36-320a8146443e" containerID="147e05257ba9a749561ca145c7600b33687653870b06942d45cac5e1c08917b5" exitCode=0
Jul 17 21:23:20.993515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:20.993145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g" event={"ID":"789287cc-65cb-4d08-9c36-320a8146443e","Type":"ContainerDied","Data":"147e05257ba9a749561ca145c7600b33687653870b06942d45cac5e1c08917b5"}
Jul 17 21:23:21.999151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:21.999112    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g" event={"ID":"789287cc-65cb-4d08-9c36-320a8146443e","Type":"ContainerStarted","Data":"5587e1359d0fc1864667f5f4b6ee35a306e8705ebea2ab110da6aab927c4edd3"}
Jul 17 21:23:21.999643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:21.999234    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:22.033423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:22.033377    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g" podStartSLOduration=2.7125170130000003 podStartE2EDuration="7.033359688s" totalImagesPullingTime="3.253642676s" totalInitContainerRuntime="1.067199999s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:22.031900304 +0000 UTC m=+378.012270782" watchObservedRunningTime="2026-07-17 21:23:22.033359688 +0000 UTC m=+378.013730165"
Jul 17 21:23:33.005512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:33.005479    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6d6dbd9688-wf99g"
Jul 17 21:23:34.306716 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.306687    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"]
Jul 17 21:23:34.309122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.309101    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.312422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312353    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 17 21:23:34.312422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312358    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 17 21:23:34.312422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312358    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 17 21:23:34.312422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312358    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-sq5l8\""
Jul 17 21:23:34.312764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312361    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 17 21:23:34.312764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.312408    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 17 21:23:34.319785 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.319764    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"]
Jul 17 21:23:34.413036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.413005    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hg9\" (UniqueName: \"kubernetes.io/projected/b5f0e76e-e33a-4fa0-82d9-af195318c546-kube-api-access-m5hg9\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.413177 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.413043    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-metrics-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.413177 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.413074    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5f0e76e-e33a-4fa0-82d9-af195318c546-manager-config\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.413177 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.413133    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.514018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.513982    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hg9\" (UniqueName: \"kubernetes.io/projected/b5f0e76e-e33a-4fa0-82d9-af195318c546-kube-api-access-m5hg9\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.514018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.514019    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-metrics-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.514267 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.514039    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5f0e76e-e33a-4fa0-82d9-af195318c546-manager-config\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.514267 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.514069    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.514869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.514787    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/b5f0e76e-e33a-4fa0-82d9-af195318c546-manager-config\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.516760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.516738    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.516861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.516739    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/b5f0e76e-e33a-4fa0-82d9-af195318c546-metrics-cert\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.529040 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.529010    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hg9\" (UniqueName: \"kubernetes.io/projected/b5f0e76e-e33a-4fa0-82d9-af195318c546-kube-api-access-m5hg9\") pod \"lws-controller-manager-76f7bc5b6f-qd4k2\" (UID: \"b5f0e76e-e33a-4fa0-82d9-af195318c546\") " pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.617954 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.617875    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:34.754782 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:34.754625    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"]
Jul 17 21:23:34.757435 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:23:34.757405    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5f0e76e_e33a_4fa0_82d9_af195318c546.slice/crio-deb6576331ef6e94b0a3abaf500c6ec86efb8a883288e105143e331b632d8444 WatchSource:0}: Error finding container deb6576331ef6e94b0a3abaf500c6ec86efb8a883288e105143e331b632d8444: Status 404 returned error can't find the container with id deb6576331ef6e94b0a3abaf500c6ec86efb8a883288e105143e331b632d8444
Jul 17 21:23:35.058309 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:35.058270    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2" event={"ID":"b5f0e76e-e33a-4fa0-82d9-af195318c546","Type":"ContainerStarted","Data":"deb6576331ef6e94b0a3abaf500c6ec86efb8a883288e105143e331b632d8444"}
Jul 17 21:23:37.069954 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:37.069908    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2" event={"ID":"b5f0e76e-e33a-4fa0-82d9-af195318c546","Type":"ContainerStarted","Data":"838448d3295eedcf19df7f95671a0165dc73636acbe25ef0b716476b5bffd763"}
Jul 17 21:23:37.070320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:37.070021    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:37.091013 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:37.090961    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2" podStartSLOduration=1.527050428 podStartE2EDuration="3.09094489s" totalImagesPullingTime="1.563894462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:37.089555559 +0000 UTC m=+393.069926038" watchObservedRunningTime="2026-07-17 21:23:37.09094489 +0000 UTC m=+393.071315435"
Jul 17 21:23:43.123370 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.123338    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:43.125559 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.125543    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.128556 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.128538    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 17 21:23:43.128705 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.128685    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 17 21:23:43.128847 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.128555    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-ddgxb\""
Jul 17 21:23:43.128847 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.128598    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 17 21:23:43.143623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.143587    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:43.282350 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282318    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282358    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282387    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282438    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4nwt\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282470    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282499    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282533    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282580    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.282694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.282616    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.383880 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383805    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.383880 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383851    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383891    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383918    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4nwt\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383950    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.383983    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384009    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384073    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384120    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384263    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384360    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384432    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384573 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384553    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.384789 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.384774    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.386277 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.386254    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.386468 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.386449    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.396592 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.396571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.397869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.397848    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4nwt\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt\") pod \"data-science-gateway-data-science-gateway-class-84cc848bd-gblpf\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.435903 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.435869    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:43.574498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:43.574472    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:43.576256 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:23:43.576223    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ac3f5b9_cc47_474a_aa82_7be3bacbdf17.slice/crio-820d2039c17c67a5aeaed9ee5c04e989d1e8b044f3f411d2a282156bf4f09651 WatchSource:0}: Error finding container 820d2039c17c67a5aeaed9ee5c04e989d1e8b044f3f411d2a282156bf4f09651: Status 404 returned error can't find the container with id 820d2039c17c67a5aeaed9ee5c04e989d1e8b044f3f411d2a282156bf4f09651
Jul 17 21:23:44.049615 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.049581    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"]
Jul 17 21:23:44.052432 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.052417    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.055819 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.055792    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 17 21:23:44.055934 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.055819    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 17 21:23:44.055934 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.055925    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 17 21:23:44.056010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.055950    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-747hd\""
Jul 17 21:23:44.067572 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.067550    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"]
Jul 17 21:23:44.099990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.099957    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" event={"ID":"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17","Type":"ContainerStarted","Data":"820d2039c17c67a5aeaed9ee5c04e989d1e8b044f3f411d2a282156bf4f09651"}
Jul 17 21:23:44.190536 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190502    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190553    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5b0183-f8a6-4173-9c16-4a91202649b0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190578    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190601    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcs2z\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-kube-api-access-zcs2z\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190672    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.190997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.190791    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291587    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291640    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291707    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291775    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5b0183-f8a6-4173-9c16-4a91202649b0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291800    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.291901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.291817    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcs2z\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-kube-api-access-zcs2z\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.293054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.293003    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.295015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.294972    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.295366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.295342    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/ad5b0183-f8a6-4173-9c16-4a91202649b0-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.295463 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.295442    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.296004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.295962    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.301812 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.301753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.302219 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.302196    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcs2z\" (UniqueName: \"kubernetes.io/projected/ad5b0183-f8a6-4173-9c16-4a91202649b0-kube-api-access-zcs2z\") pod \"istiod-openshift-gateway-94bb8fbfd-gzv5k\" (UID: \"ad5b0183-f8a6-4173-9c16-4a91202649b0\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.362081 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.361915    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:44.743264 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:44.743234    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"]
Jul 17 21:23:44.745327 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:23:44.745289    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad5b0183_f8a6_4173_9c16_4a91202649b0.slice/crio-20eff54a9c52cb52595693fa9c7a59799817946fd2de9365af1071647e9ac6a9 WatchSource:0}: Error finding container 20eff54a9c52cb52595693fa9c7a59799817946fd2de9365af1071647e9ac6a9: Status 404 returned error can't find the container with id 20eff54a9c52cb52595693fa9c7a59799817946fd2de9365af1071647e9ac6a9
Jul 17 21:23:45.108600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:45.108566    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k" event={"ID":"ad5b0183-f8a6-4173-9c16-4a91202649b0","Type":"ContainerStarted","Data":"20eff54a9c52cb52595693fa9c7a59799817946fd2de9365af1071647e9ac6a9"}
Jul 17 21:23:47.189035 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:47.188985    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:23:47.189439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:47.189080    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:23:47.189439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:47.189127    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:23:47.189533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:47.189469    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:23:47.189533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:47.189523    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:23:48.075769 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.075716    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 21:23:48.123184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.123154    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k" event={"ID":"ad5b0183-f8a6-4173-9c16-4a91202649b0","Type":"ContainerStarted","Data":"1f13da5f1f5d869487aa5bc7e58d1ada864ae393a00c76a85ec035cdf5617c15"}
Jul 17 21:23:48.123370 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.123310    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:48.124822 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.124798    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" event={"ID":"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17","Type":"ContainerStarted","Data":"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"}
Jul 17 21:23:48.125061 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.125040    2546 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-gzv5k container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:23:48.125131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.125086    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k" podUID="ad5b0183-f8a6-4173-9c16-4a91202649b0" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:23:48.222661 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.222605    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k" podStartSLOduration=1.781132368 podStartE2EDuration="4.222584959s" totalImagesPullingTime="2.441452591s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:48.219277521 +0000 UTC m=+404.199647999" watchObservedRunningTime="2026-07-17 21:23:48.222584959 +0000 UTC m=+404.202955435"
Jul 17 21:23:48.278275 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.278219    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" podStartSLOduration=1.667649978 podStartE2EDuration="5.278201583s" totalImagesPullingTime="3.610551605s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:23:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:23:48.277029876 +0000 UTC m=+404.257400353" watchObservedRunningTime="2026-07-17 21:23:48.278201583 +0000 UTC m=+404.258572150"
Jul 17 21:23:48.436355 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.436272    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:48.437960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.437925    2546 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Jul 17 21:23:48.438068 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.437971    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Jul 17 21:23:48.496850 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:48.496826    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:49.129404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:49.129376    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-gzv5k"
Jul 17 21:23:49.436936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:49.436863    2546 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused" start-of-body=
Jul 17 21:23:49.436936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:49.436920    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.26:15021/healthz/ready\": dial tcp 10.133.0.26:15021: connect: connection refused"
Jul 17 21:23:50.132936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:50.132879    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy" containerID="cri-o://4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137" gracePeriod=30
Jul 17 21:23:55.388491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.388461    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:55.482420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482328    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482371    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482406    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482431    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482466    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482500    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4nwt\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482542    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482562    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.482696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482621    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy\") pod \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\" (UID: \"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17\") "
Jul 17 21:23:55.483009 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482813    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:55.483009 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482822    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:55.483009 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482868    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:55.483009 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482964    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:23:55.483009 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.482972    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data" (OuterVolumeSpecName: "istio-data") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:55.484876 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.484843    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt" (OuterVolumeSpecName: "kube-api-access-m4nwt") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "kube-api-access-m4nwt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:23:55.484985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.484872    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token" (OuterVolumeSpecName: "istio-token") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:23:55.484985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.484904    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:23:55.484985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.484940    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" (UID: "4ac3f5b9-cc47-474a-aa82-7be3bacbdf17"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:23:55.584101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584048    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584103    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584117    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584130    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584144    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584155    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584168    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4nwt\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-kube-api-access-m4nwt\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584182    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:55.584331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:55.584195    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:23:56.158443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.158415    2546 generic.go:356] "Generic (PLEG): container finished" podID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerID="4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137" exitCode=0
Jul 17 21:23:56.158590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.158460    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"
Jul 17 21:23:56.158590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.158464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" event={"ID":"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17","Type":"ContainerDied","Data":"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"}
Jul 17 21:23:56.158672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.158600    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf" event={"ID":"4ac3f5b9-cc47-474a-aa82-7be3bacbdf17","Type":"ContainerDied","Data":"820d2039c17c67a5aeaed9ee5c04e989d1e8b044f3f411d2a282156bf4f09651"}
Jul 17 21:23:56.158672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.158629    2546 scope.go:117] "RemoveContainer" containerID="4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"
Jul 17 21:23:56.167573 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.167558    2546 scope.go:117] "RemoveContainer" containerID="4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"
Jul 17 21:23:56.167911 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:23:56.167892    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137\": container with ID starting with 4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137 not found: ID does not exist" containerID="4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"
Jul 17 21:23:56.167984 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.167917    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137"} err="failed to get container status \"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137\": rpc error: code = NotFound desc = could not find container \"4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137\": container with ID starting with 4f1ed107ab9028cce1bb37f81672f74f97f6e84d1db7e84fb45b194631f40137 not found: ID does not exist"
Jul 17 21:23:56.179645 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.179622    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:56.183344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.183325    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-84cc848bd-gblpf"]
Jul 17 21:23:56.536083 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:23:56.536049    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" path="/var/lib/kubelet/pods/4ac3f5b9-cc47-474a-aa82-7be3bacbdf17/volumes"
Jul 17 21:24:51.941293 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.941251    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"]
Jul 17 21:24:51.941784 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.941560    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy"
Jul 17 21:24:51.941784 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.941572    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy"
Jul 17 21:24:51.941784 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.941628    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ac3f5b9-cc47-474a-aa82-7be3bacbdf17" containerName="istio-proxy"
Jul 17 21:24:51.943742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.943705    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:51.945995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.945972    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mn5tn\""
Jul 17 21:24:51.946077 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.945997    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:24:51.946740 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.946705    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:24:51.953522 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:51.953500    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"]
Jul 17 21:24:52.097737 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.097695    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.097943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.097791    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t447c\" (UniqueName: \"kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.097943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.097842    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.198948 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.198845    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.198948 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.198942    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.199187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.198966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t447c\" (UniqueName: \"kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.199247 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.199221    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.199337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.199317    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.207889 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.207859    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t447c\" (UniqueName: \"kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.253358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.253328    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:52.385618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.385541    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"]
Jul 17 21:24:52.388376 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:24:52.388352    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c7df81c_55a3_4132_9247_a6befdd7928f.slice/crio-6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0 WatchSource:0}: Error finding container 6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0: Status 404 returned error can't find the container with id 6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0
Jul 17 21:24:52.401260 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.401231    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f" event={"ID":"2c7df81c-55a3-4132-9247-a6befdd7928f","Type":"ContainerStarted","Data":"6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0"}
Jul 17 21:24:52.550330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.550299    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"]
Jul 17 21:24:52.552794 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.552778    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.563041 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.563016    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"]
Jul 17 21:24:52.702773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.702709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.702930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.702816    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.702930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.702848    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6p5zv\" (UniqueName: \"kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.803354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.803313    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.803538 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.803359    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6p5zv\" (UniqueName: \"kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.803538 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.803420    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.803704 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.803683    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.803825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.803785    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.813270 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.813253    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6p5zv\" (UniqueName: \"kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:52.861395 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:52.861366    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:53.007771 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.007748    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"]
Jul 17 21:24:53.009448 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:24:53.009426    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee99d263_71ca_4bc9_b145_423943857726.slice/crio-fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35 WatchSource:0}: Error finding container fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35: Status 404 returned error can't find the container with id fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35
Jul 17 21:24:53.134235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.134203    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"]
Jul 17 21:24:53.136279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.136265    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.151344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.151233    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"]
Jul 17 21:24:53.307550 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.307450    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.307550 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.307497    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.307550 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.307517    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7f4p\" (UniqueName: \"kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.407420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.407384    2546 generic.go:356] "Generic (PLEG): container finished" podID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerID="56ce4e30f9a051400106de47d6ba6393ea90e47b459587a0db8ef49ba7002772" exitCode=0
Jul 17 21:24:53.407571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.407477    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f" event={"ID":"2c7df81c-55a3-4132-9247-a6befdd7928f","Type":"ContainerDied","Data":"56ce4e30f9a051400106de47d6ba6393ea90e47b459587a0db8ef49ba7002772"}
Jul 17 21:24:53.407958 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.407933    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.408074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.407991    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.408074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.408023    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7f4p\" (UniqueName: \"kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.408376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.408354    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.408472 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.408396    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.409153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.409132    2546 generic.go:356] "Generic (PLEG): container finished" podID="ee99d263-71ca-4bc9-b145-423943857726" containerID="10842b29a23c5255eba187195b69168cc25b15d733f08e4793d38b2e67cf5f22" exitCode=0
Jul 17 21:24:53.409240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.409174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc" event={"ID":"ee99d263-71ca-4bc9-b145-423943857726","Type":"ContainerDied","Data":"10842b29a23c5255eba187195b69168cc25b15d733f08e4793d38b2e67cf5f22"}
Jul 17 21:24:53.409240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.409214    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc" event={"ID":"ee99d263-71ca-4bc9-b145-423943857726","Type":"ContainerStarted","Data":"fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35"}
Jul 17 21:24:53.432191 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.432156    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7f4p\" (UniqueName: \"kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.461790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.461762    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:53.543117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.543083    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"]
Jul 17 21:24:53.546188 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.546163    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.557306 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.557279    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"]
Jul 17 21:24:53.600513 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.600491    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"]
Jul 17 21:24:53.602502 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:24:53.602477    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ed2b81_1c86_4721_9ab5_d248cc4a053b.slice/crio-3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2 WatchSource:0}: Error finding container 3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2: Status 404 returned error can't find the container with id 3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2
Jul 17 21:24:53.710234 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.710197    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.710382 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.710248    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.710382 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.710286    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46b4x\" (UniqueName: \"kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.810883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.810849    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46b4x\" (UniqueName: \"kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.811082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.810900    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.811082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.810937    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.811276 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.811261    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.811332 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.811296    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.819837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.819812    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46b4x\" (UniqueName: \"kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.859035 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.858976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:53.987565 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:53.987540    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"]
Jul 17 21:24:54.016445 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:24:54.016415    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5634c3de_f7cb_4da5_beff_bf4ccbc0215a.slice/crio-26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab WatchSource:0}: Error finding container 26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab: Status 404 returned error can't find the container with id 26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab
Jul 17 21:24:54.415425 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.415352    2546 generic.go:356] "Generic (PLEG): container finished" podID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerID="96e902fa4811488dabb40bdf408b6828267f83677ea9c27a249183a0f7924a84" exitCode=0
Jul 17 21:24:54.415425 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.415408    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f" event={"ID":"2c7df81c-55a3-4132-9247-a6befdd7928f","Type":"ContainerDied","Data":"96e902fa4811488dabb40bdf408b6828267f83677ea9c27a249183a0f7924a84"}
Jul 17 21:24:54.417304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.417284    2546 generic.go:356] "Generic (PLEG): container finished" podID="ee99d263-71ca-4bc9-b145-423943857726" containerID="b065ee675e56df6378c4e760bf5bdc534e23d266a039c61496dfeb64ed892f7c" exitCode=0
Jul 17 21:24:54.417384 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.417346    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc" event={"ID":"ee99d263-71ca-4bc9-b145-423943857726","Type":"ContainerDied","Data":"b065ee675e56df6378c4e760bf5bdc534e23d266a039c61496dfeb64ed892f7c"}
Jul 17 21:24:54.419114 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.419089    2546 generic.go:356] "Generic (PLEG): container finished" podID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerID="f4f2a57e3c364400444267d38f9bb5d4721c4d33dc5e42764e7bfd4366f6ba80" exitCode=0
Jul 17 21:24:54.419234 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.419215    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2" event={"ID":"5634c3de-f7cb-4da5-beff-bf4ccbc0215a","Type":"ContainerDied","Data":"f4f2a57e3c364400444267d38f9bb5d4721c4d33dc5e42764e7bfd4366f6ba80"}
Jul 17 21:24:54.419394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.419249    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2" event={"ID":"5634c3de-f7cb-4da5-beff-bf4ccbc0215a","Type":"ContainerStarted","Data":"26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab"}
Jul 17 21:24:54.420821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.420796    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerID="113306d726273f5fcbf80a4a67d3df79e8a19ebd88e4e546f9ab555a312a9aa9" exitCode=0
Jul 17 21:24:54.420900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.420834    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn" event={"ID":"d0ed2b81-1c86-4721-9ab5-d248cc4a053b","Type":"ContainerDied","Data":"113306d726273f5fcbf80a4a67d3df79e8a19ebd88e4e546f9ab555a312a9aa9"}
Jul 17 21:24:54.420900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:54.420869    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn" event={"ID":"d0ed2b81-1c86-4721-9ab5-d248cc4a053b","Type":"ContainerStarted","Data":"3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2"}
Jul 17 21:24:55.427371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.427340    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerID="eda50d2fd648fc566b2049c76ba0117d79c7ab2d1e7163da23b152a02a50fde3" exitCode=0
Jul 17 21:24:55.427854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.427419    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn" event={"ID":"d0ed2b81-1c86-4721-9ab5-d248cc4a053b","Type":"ContainerDied","Data":"eda50d2fd648fc566b2049c76ba0117d79c7ab2d1e7163da23b152a02a50fde3"}
Jul 17 21:24:55.429684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.429661    2546 generic.go:356] "Generic (PLEG): container finished" podID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerID="5afc84638410834135bf066ab529a0bf937bdac8072cd462b271b9fcac5b972c" exitCode=0
Jul 17 21:24:55.429791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.429741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f" event={"ID":"2c7df81c-55a3-4132-9247-a6befdd7928f","Type":"ContainerDied","Data":"5afc84638410834135bf066ab529a0bf937bdac8072cd462b271b9fcac5b972c"}
Jul 17 21:24:55.436477 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.434376    2546 generic.go:356] "Generic (PLEG): container finished" podID="ee99d263-71ca-4bc9-b145-423943857726" containerID="d1bb26bb3f49d1585b7d661169aa245ba412cb19c22e7196974eb7b7de51b9c2" exitCode=0
Jul 17 21:24:55.436477 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.434488    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc" event={"ID":"ee99d263-71ca-4bc9-b145-423943857726","Type":"ContainerDied","Data":"d1bb26bb3f49d1585b7d661169aa245ba412cb19c22e7196974eb7b7de51b9c2"}
Jul 17 21:24:55.438575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.438555    2546 generic.go:356] "Generic (PLEG): container finished" podID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerID="6650efe3bf1e529756b087220a63e1a39323ba1a83652936c10606fc1a1344d9" exitCode=0
Jul 17 21:24:55.438666 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:55.438583    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2" event={"ID":"5634c3de-f7cb-4da5-beff-bf4ccbc0215a","Type":"ContainerDied","Data":"6650efe3bf1e529756b087220a63e1a39323ba1a83652936c10606fc1a1344d9"}
Jul 17 21:24:56.445304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.445265    2546 generic.go:356] "Generic (PLEG): container finished" podID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerID="f87a9a4081a3e4d5455b3d74c1bff11513b0daae6ffa3f0fb5df7db8181913d8" exitCode=0
Jul 17 21:24:56.445697 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.445342    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2" event={"ID":"5634c3de-f7cb-4da5-beff-bf4ccbc0215a","Type":"ContainerDied","Data":"f87a9a4081a3e4d5455b3d74c1bff11513b0daae6ffa3f0fb5df7db8181913d8"}
Jul 17 21:24:56.447494 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.447471    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerID="2765f392fb7d67be7adb7566936a329d1712d0da4b1a48b9290969936fa9ee80" exitCode=0
Jul 17 21:24:56.447644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.447541    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn" event={"ID":"d0ed2b81-1c86-4721-9ab5-d248cc4a053b","Type":"ContainerDied","Data":"2765f392fb7d67be7adb7566936a329d1712d0da4b1a48b9290969936fa9ee80"}
Jul 17 21:24:56.596992 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.596968    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:56.599771 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.599754    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:56.734332 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734299    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t447c\" (UniqueName: \"kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c\") pod \"2c7df81c-55a3-4132-9247-a6befdd7928f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") "
Jul 17 21:24:56.734332 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734330    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle\") pod \"ee99d263-71ca-4bc9-b145-423943857726\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") "
Jul 17 21:24:56.734512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734358    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle\") pod \"2c7df81c-55a3-4132-9247-a6befdd7928f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") "
Jul 17 21:24:56.734512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734388    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util\") pod \"2c7df81c-55a3-4132-9247-a6befdd7928f\" (UID: \"2c7df81c-55a3-4132-9247-a6befdd7928f\") "
Jul 17 21:24:56.734512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734408    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util\") pod \"ee99d263-71ca-4bc9-b145-423943857726\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") "
Jul 17 21:24:56.734512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734423    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6p5zv\" (UniqueName: \"kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv\") pod \"ee99d263-71ca-4bc9-b145-423943857726\" (UID: \"ee99d263-71ca-4bc9-b145-423943857726\") "
Jul 17 21:24:56.734956 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734925    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle" (OuterVolumeSpecName: "bundle") pod "2c7df81c-55a3-4132-9247-a6befdd7928f" (UID: "2c7df81c-55a3-4132-9247-a6befdd7928f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:56.735070 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.734957    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle" (OuterVolumeSpecName: "bundle") pod "ee99d263-71ca-4bc9-b145-423943857726" (UID: "ee99d263-71ca-4bc9-b145-423943857726"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:56.736837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.736808    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv" (OuterVolumeSpecName: "kube-api-access-6p5zv") pod "ee99d263-71ca-4bc9-b145-423943857726" (UID: "ee99d263-71ca-4bc9-b145-423943857726"). InnerVolumeSpecName "kube-api-access-6p5zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:24:56.736924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.736897    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c" (OuterVolumeSpecName: "kube-api-access-t447c") pod "2c7df81c-55a3-4132-9247-a6befdd7928f" (UID: "2c7df81c-55a3-4132-9247-a6befdd7928f"). InnerVolumeSpecName "kube-api-access-t447c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:24:56.739708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.739689    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util" (OuterVolumeSpecName: "util") pod "2c7df81c-55a3-4132-9247-a6befdd7928f" (UID: "2c7df81c-55a3-4132-9247-a6befdd7928f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:56.740853 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.740819    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util" (OuterVolumeSpecName: "util") pod "ee99d263-71ca-4bc9-b145-423943857726" (UID: "ee99d263-71ca-4bc9-b145-423943857726"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835893    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t447c\" (UniqueName: \"kubernetes.io/projected/2c7df81c-55a3-4132-9247-a6befdd7928f-kube-api-access-t447c\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835916    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835926    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835933    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/2c7df81c-55a3-4132-9247-a6befdd7928f-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835940    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/ee99d263-71ca-4bc9-b145-423943857726-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:56.835947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:56.835948    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6p5zv\" (UniqueName: \"kubernetes.io/projected/ee99d263-71ca-4bc9-b145-423943857726-kube-api-access-6p5zv\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.453532 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.453506    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f"
Jul 17 21:24:57.454036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.453506    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f" event={"ID":"2c7df81c-55a3-4132-9247-a6befdd7928f","Type":"ContainerDied","Data":"6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0"}
Jul 17 21:24:57.454036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.453628    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6bbe98f7fedaa8887fadb6b1b016d7639c499b9f63c29a2181f6d2143af935f0"
Jul 17 21:24:57.455411 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.455391    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc"
Jul 17 21:24:57.455548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.455416    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc" event={"ID":"ee99d263-71ca-4bc9-b145-423943857726","Type":"ContainerDied","Data":"fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35"}
Jul 17 21:24:57.455548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.455455    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fd5d690146130d9bf2a061f96145a3b8af37546973eedddd0681ed5b20df1e35"
Jul 17 21:24:57.606606 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.606578    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:57.608563 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.608546    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:57.743401 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743370    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46b4x\" (UniqueName: \"kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x\") pod \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") "
Jul 17 21:24:57.743587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743429    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util\") pod \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") "
Jul 17 21:24:57.743587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743452    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle\") pod \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") "
Jul 17 21:24:57.743587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743493    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7f4p\" (UniqueName: \"kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p\") pod \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") "
Jul 17 21:24:57.743587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743511    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle\") pod \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\" (UID: \"d0ed2b81-1c86-4721-9ab5-d248cc4a053b\") "
Jul 17 21:24:57.743587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.743544    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util\") pod \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\" (UID: \"5634c3de-f7cb-4da5-beff-bf4ccbc0215a\") "
Jul 17 21:24:57.744072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.744046    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle" (OuterVolumeSpecName: "bundle") pod "d0ed2b81-1c86-4721-9ab5-d248cc4a053b" (UID: "d0ed2b81-1c86-4721-9ab5-d248cc4a053b"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:57.744168 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.744135    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle" (OuterVolumeSpecName: "bundle") pod "5634c3de-f7cb-4da5-beff-bf4ccbc0215a" (UID: "5634c3de-f7cb-4da5-beff-bf4ccbc0215a"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:57.745760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.745736    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x" (OuterVolumeSpecName: "kube-api-access-46b4x") pod "5634c3de-f7cb-4da5-beff-bf4ccbc0215a" (UID: "5634c3de-f7cb-4da5-beff-bf4ccbc0215a"). InnerVolumeSpecName "kube-api-access-46b4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:24:57.746091 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.746069    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p" (OuterVolumeSpecName: "kube-api-access-t7f4p") pod "d0ed2b81-1c86-4721-9ab5-d248cc4a053b" (UID: "d0ed2b81-1c86-4721-9ab5-d248cc4a053b"). InnerVolumeSpecName "kube-api-access-t7f4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:24:57.749419 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.749380    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util" (OuterVolumeSpecName: "util") pod "5634c3de-f7cb-4da5-beff-bf4ccbc0215a" (UID: "5634c3de-f7cb-4da5-beff-bf4ccbc0215a"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:57.750552 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.750517    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util" (OuterVolumeSpecName: "util") pod "d0ed2b81-1c86-4721-9ab5-d248cc4a053b" (UID: "d0ed2b81-1c86-4721-9ab5-d248cc4a053b"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:24:57.844904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844864    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.844904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844899    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.844904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844909    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7f4p\" (UniqueName: \"kubernetes.io/projected/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-kube-api-access-t7f4p\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.845131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844919    2546 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/d0ed2b81-1c86-4721-9ab5-d248cc4a053b-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.845131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844927    2546 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-util\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:57.845131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:57.844935    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46b4x\" (UniqueName: \"kubernetes.io/projected/5634c3de-f7cb-4da5-beff-bf4ccbc0215a-kube-api-access-46b4x\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:24:58.466256 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.466226    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn"
Jul 17 21:24:58.466758 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.466260    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn" event={"ID":"d0ed2b81-1c86-4721-9ab5-d248cc4a053b","Type":"ContainerDied","Data":"3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2"}
Jul 17 21:24:58.466758 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.466328    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3f2d1f5d8b513b5e75e27924eb10e39dda531c2d1acc097700c40d84fb4718d2"
Jul 17 21:24:58.468171 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.468135    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2" event={"ID":"5634c3de-f7cb-4da5-beff-bf4ccbc0215a","Type":"ContainerDied","Data":"26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab"}
Jul 17 21:24:58.468171 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.468149    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2"
Jul 17 21:24:58.468301 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:24:58.468176    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26eb21cf5b8dfb46dda6eae5c6a27651bc9b15dd4a06c1ecfea876c0908caaab"
Jul 17 21:25:16.164837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:16.164801    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:25:41.185512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.185458    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-9b79c56dc-rswbn" podUID="0811e21b-2d9e-4f7e-b588-29561e7ace39" containerName="console" containerID="cri-o://f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40" gracePeriod=15
Jul 17 21:25:41.425755 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.425714    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468582    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468632    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468674    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468738    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468798    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468823    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.468865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.468846    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9k2lx\" (UniqueName: \"kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx\") pod \"0811e21b-2d9e-4f7e-b588-29561e7ace39\" (UID: \"0811e21b-2d9e-4f7e-b588-29561e7ace39\") "
Jul 17 21:25:41.469313 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.469241    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config" (OuterVolumeSpecName: "console-config") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:25:41.469313 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.469264    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:25:41.469556 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.469529    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:25:41.469706 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.469676    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca" (OuterVolumeSpecName: "service-ca") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:25:41.471603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.471567    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx" (OuterVolumeSpecName: "kube-api-access-9k2lx") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "kube-api-access-9k2lx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:25:41.471603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.471577    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:25:41.471849 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.471636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "0811e21b-2d9e-4f7e-b588-29561e7ace39" (UID: "0811e21b-2d9e-4f7e-b588-29561e7ace39"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:25:41.570253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570222    2546 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-oauth-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570250    2546 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-serving-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570260    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9k2lx\" (UniqueName: \"kubernetes.io/projected/0811e21b-2d9e-4f7e-b588-29561e7ace39-kube-api-access-9k2lx\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570270    2546 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570278    2546 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-trusted-ca-bundle\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570286    2546 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.570473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.570293    2546 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/0811e21b-2d9e-4f7e-b588-29561e7ace39-console-oauth-config\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:25:41.662830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.662795    2546 generic.go:356] "Generic (PLEG): container finished" podID="0811e21b-2d9e-4f7e-b588-29561e7ace39" containerID="f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40" exitCode=0
Jul 17 21:25:41.662985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.662844    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-9b79c56dc-rswbn"
Jul 17 21:25:41.662985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.662868    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b79c56dc-rswbn" event={"ID":"0811e21b-2d9e-4f7e-b588-29561e7ace39","Type":"ContainerDied","Data":"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"}
Jul 17 21:25:41.662985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.662923    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-9b79c56dc-rswbn" event={"ID":"0811e21b-2d9e-4f7e-b588-29561e7ace39","Type":"ContainerDied","Data":"89e7d7dfc919089255fe1381892dc438a0370e0d563a7fa1ded477665da7ac24"}
Jul 17 21:25:41.662985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.662947    2546 scope.go:117] "RemoveContainer" containerID="f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"
Jul 17 21:25:41.672116 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.672099    2546 scope.go:117] "RemoveContainer" containerID="f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"
Jul 17 21:25:41.672359 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:25:41.672341    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40\": container with ID starting with f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40 not found: ID does not exist" containerID="f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"
Jul 17 21:25:41.672422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.672365    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40"} err="failed to get container status \"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40\": rpc error: code = NotFound desc = could not find container \"f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40\": container with ID starting with f161b2014b60b993a2537b8663bc40b8f0ff37d623328a079a97ed0089d9ac40 not found: ID does not exist"
Jul 17 21:25:41.685602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.685577    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:25:41.690547 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:41.690523    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-9b79c56dc-rswbn"]
Jul 17 21:25:42.537298 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:25:42.537266    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0811e21b-2d9e-4f7e-b588-29561e7ace39" path="/var/lib/kubelet/pods/0811e21b-2d9e-4f7e-b588-29561e7ace39/volumes"
Jul 17 21:26:13.085685 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085648    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085937    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085950    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085959    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085965    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085976    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085981    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085989    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0811e21b-2d9e-4f7e-b588-29561e7ace39" containerName="console"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085994    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0811e21b-2d9e-4f7e-b588-29561e7ace39" containerName="console"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.085999    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086004    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086009    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086014    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086019    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086024    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086029    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086034    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086041    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086045    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086051    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086056    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="util"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086062    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086066    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086072    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086076    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086081    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086085    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="pull"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086131    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee99d263-71ca-4bc9-b145-423943857726" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086138    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0811e21b-2d9e-4f7e-b588-29561e7ace39" containerName="console"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086143    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2c7df81c-55a3-4132-9247-a6befdd7928f" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086149    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5634c3de-f7cb-4da5-beff-bf4ccbc0215a" containerName="extract"
Jul 17 21:26:13.086148 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.086154    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ed2b81-1c86-4721-9ab5-d248cc4a053b" containerName="extract"
Jul 17 21:26:13.092147 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.091489    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.094145 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.094114    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:26:13.094145 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.094133    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 17 21:26:13.094145 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.094114    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:26:13.095213 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.095182    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-mn5tn\""
Jul 17 21:26:13.097529 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.097508    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:13.183173 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.183138    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:13.202036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.202007    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.202179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.202047    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.303285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.303257    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.303437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.303293    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.303894 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.303875    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.312471 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.312445    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr\") pod \"limitador-limitador-64c94fd64f-klx8k\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.403434 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.403376    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:13.540227 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.540202    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:13.542047 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:26:13.542014    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5da643e3_ad14_45c9_876b_a192874efa50.slice/crio-b79bfff4cd19a0dfa67849e47640dd2fd6842bb0c4e93a80a3ab6fd543c91e4d WatchSource:0}: Error finding container b79bfff4cd19a0dfa67849e47640dd2fd6842bb0c4e93a80a3ab6fd543c91e4d: Status 404 returned error can't find the container with id b79bfff4cd19a0dfa67849e47640dd2fd6842bb0c4e93a80a3ab6fd543c91e4d
Jul 17 21:26:13.805454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:13.805424    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" event={"ID":"5da643e3-ad14-45c9-876b-a192874efa50","Type":"ContainerStarted","Data":"b79bfff4cd19a0dfa67849e47640dd2fd6842bb0c4e93a80a3ab6fd543c91e4d"}
Jul 17 21:26:16.821812 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:16.821774    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" event={"ID":"5da643e3-ad14-45c9-876b-a192874efa50","Type":"ContainerStarted","Data":"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"}
Jul 17 21:26:16.822293 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:16.821859    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:16.848315 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:16.848271    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" podStartSLOduration=1.320929335 podStartE2EDuration="3.848258929s" totalImagesPullingTime="2.527329594s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:26:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:26:16.846495656 +0000 UTC m=+552.826866134" watchObservedRunningTime="2026-07-17 21:26:16.848258929 +0000 UTC m=+552.828629405"
Jul 17 21:26:27.828410 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:27.828378    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:28.202166 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.202065    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:28.202411 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.202297    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" podUID="5da643e3-ad14-45c9-876b-a192874efa50" containerName="limitador" containerID="cri-o://ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc" gracePeriod=30
Jul 17 21:26:28.743946 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.743923    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:28.807803 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.807703    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file\") pod \"5da643e3-ad14-45c9-876b-a192874efa50\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") "
Jul 17 21:26:28.807803 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.807766    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr\") pod \"5da643e3-ad14-45c9-876b-a192874efa50\" (UID: \"5da643e3-ad14-45c9-876b-a192874efa50\") "
Jul 17 21:26:28.808055 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.808029    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file" (OuterVolumeSpecName: "config-file") pod "5da643e3-ad14-45c9-876b-a192874efa50" (UID: "5da643e3-ad14-45c9-876b-a192874efa50"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:26:28.809895 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.809862    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr" (OuterVolumeSpecName: "kube-api-access-2z7dr") pod "5da643e3-ad14-45c9-876b-a192874efa50" (UID: "5da643e3-ad14-45c9-876b-a192874efa50"). InnerVolumeSpecName "kube-api-access-2z7dr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:26:28.881396 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.881362    2546 generic.go:356] "Generic (PLEG): container finished" podID="5da643e3-ad14-45c9-876b-a192874efa50" containerID="ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc" exitCode=0
Jul 17 21:26:28.881818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.881398    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k"
Jul 17 21:26:28.881818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.881484    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" event={"ID":"5da643e3-ad14-45c9-876b-a192874efa50","Type":"ContainerDied","Data":"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"}
Jul 17 21:26:28.881818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.881537    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-klx8k" event={"ID":"5da643e3-ad14-45c9-876b-a192874efa50","Type":"ContainerDied","Data":"b79bfff4cd19a0dfa67849e47640dd2fd6842bb0c4e93a80a3ab6fd543c91e4d"}
Jul 17 21:26:28.881818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.881559    2546 scope.go:117] "RemoveContainer" containerID="ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"
Jul 17 21:26:28.890435 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.890420    2546 scope.go:117] "RemoveContainer" containerID="ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"
Jul 17 21:26:28.890679 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:26:28.890663    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc\": container with ID starting with ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc not found: ID does not exist" containerID="ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"
Jul 17 21:26:28.890754 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.890686    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc"} err="failed to get container status \"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc\": rpc error: code = NotFound desc = could not find container \"ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc\": container with ID starting with ccbb35d2aaae73a9550fa0c929d64b1e2ee012e561def79825a2391b2f47c5cc not found: ID does not exist"
Jul 17 21:26:28.902176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.902148    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:28.908390 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.908362    2546 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5da643e3-ad14-45c9-876b-a192874efa50-config-file\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:26:28.908563 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.908522    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z7dr\" (UniqueName: \"kubernetes.io/projected/5da643e3-ad14-45c9-876b-a192874efa50-kube-api-access-2z7dr\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:26:28.910064 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:28.910046    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-klx8k"]
Jul 17 21:26:29.578258 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.578231    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"]
Jul 17 21:26:29.578523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.578512    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5da643e3-ad14-45c9-876b-a192874efa50" containerName="limitador"
Jul 17 21:26:29.578564 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.578524    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5da643e3-ad14-45c9-876b-a192874efa50" containerName="limitador"
Jul 17 21:26:29.578605 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.578575    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5da643e3-ad14-45c9-876b-a192874efa50" containerName="limitador"
Jul 17 21:26:29.589670 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.589642    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.590944 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.590916    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"]
Jul 17 21:26:29.592097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.592080    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-6gspf\""
Jul 17 21:26:29.592188 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.592139    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 17 21:26:29.592257 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.592091    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 17 21:26:29.592918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.592898    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 17 21:26:29.716036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.716003    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f6fb0ce-2931-4750-ab12-ca50960201bb-data\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.716036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.716040    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpnbm\" (UniqueName: \"kubernetes.io/projected/7f6fb0ce-2931-4750-ab12-ca50960201bb-kube-api-access-xpnbm\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.817289 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.817246    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpnbm\" (UniqueName: \"kubernetes.io/projected/7f6fb0ce-2931-4750-ab12-ca50960201bb-kube-api-access-xpnbm\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.817466 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.817342    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f6fb0ce-2931-4750-ab12-ca50960201bb-data\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.817644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.817630    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/7f6fb0ce-2931-4750-ab12-ca50960201bb-data\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.826101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.826067    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpnbm\" (UniqueName: \"kubernetes.io/projected/7f6fb0ce-2931-4750-ab12-ca50960201bb-kube-api-access-xpnbm\") pod \"postgres-69fb4594cf-gz6fk\" (UID: \"7f6fb0ce-2931-4750-ab12-ca50960201bb\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:29.900808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:29.900741    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:30.045337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:30.045286    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"]
Jul 17 21:26:30.047674 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:26:30.047646    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f6fb0ce_2931_4750_ab12_ca50960201bb.slice/crio-2e97632feef84143d4fbbb17ed0551381e887ff3fe348da6c5fefed49e46bb4e WatchSource:0}: Error finding container 2e97632feef84143d4fbbb17ed0551381e887ff3fe348da6c5fefed49e46bb4e: Status 404 returned error can't find the container with id 2e97632feef84143d4fbbb17ed0551381e887ff3fe348da6c5fefed49e46bb4e
Jul 17 21:26:30.536257 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:30.536221    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5da643e3-ad14-45c9-876b-a192874efa50" path="/var/lib/kubelet/pods/5da643e3-ad14-45c9-876b-a192874efa50/volumes"
Jul 17 21:26:30.896309 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:30.896202    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk" event={"ID":"7f6fb0ce-2931-4750-ab12-ca50960201bb","Type":"ContainerStarted","Data":"2e97632feef84143d4fbbb17ed0551381e887ff3fe348da6c5fefed49e46bb4e"}
Jul 17 21:26:34.915767 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:34.915712    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk" event={"ID":"7f6fb0ce-2931-4750-ab12-ca50960201bb","Type":"ContainerStarted","Data":"095ffbb56e9bcf65b6ca306bcaac1b0f6d07c13c9b53fbaae9dc8cd5fb397e20"}
Jul 17 21:26:34.916143 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:34.915837    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:34.931897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:34.931847    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk" podStartSLOduration=1.276623103 podStartE2EDuration="5.931831093s" totalImagesPullingTime="4.65520799s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:26:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:26:34.930693381 +0000 UTC m=+570.911063860" watchObservedRunningTime="2026-07-17 21:26:34.931831093 +0000 UTC m=+570.912201604"
Jul 17 21:26:40.950510 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:40.950482    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-gz6fk"
Jul 17 21:26:41.980134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.980095    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:26:41.984673 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.984655    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:41.988900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.988879    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 17 21:26:41.989014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.988881    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-rlptf\""
Jul 17 21:26:41.989014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.988882    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 17 21:26:41.989014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.988884    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 17 21:26:41.994821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:41.994797    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:26:42.118254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.118216    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcts8\" (UniqueName: \"kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.118449 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.118271    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.219497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.219451    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcts8\" (UniqueName: \"kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.219657 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.219528    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.222029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.222012    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.228559 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.228537    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcts8\" (UniqueName: \"kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8\") pod \"authorino-798db8ff7d-pwgjw\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") " pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.294402 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.294369    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:26:42.442936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.442892    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:26:42.445028 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:26:42.445004    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8a1d39c_8012_42c0_8741_2bbd64c081a6.slice/crio-240377f41ee6e081b2eccd1a1fdecad048fcdfda48313e5c6c715b5eaf47a3d5 WatchSource:0}: Error finding container 240377f41ee6e081b2eccd1a1fdecad048fcdfda48313e5c6c715b5eaf47a3d5: Status 404 returned error can't find the container with id 240377f41ee6e081b2eccd1a1fdecad048fcdfda48313e5c6c715b5eaf47a3d5
Jul 17 21:26:42.951792 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:42.951760    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" event={"ID":"c8a1d39c-8012-42c0-8741-2bbd64c081a6","Type":"ContainerStarted","Data":"240377f41ee6e081b2eccd1a1fdecad048fcdfda48313e5c6c715b5eaf47a3d5"}
Jul 17 21:26:46.974443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:46.974402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" event={"ID":"c8a1d39c-8012-42c0-8741-2bbd64c081a6","Type":"ContainerStarted","Data":"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"}
Jul 17 21:26:46.989104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:26:46.989053    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" podStartSLOduration=2.528665755 podStartE2EDuration="5.989038918s" totalImagesPullingTime="3.460373163s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:26:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:26:46.988469704 +0000 UTC m=+582.968840180" watchObservedRunningTime="2026-07-17 21:26:46.989038918 +0000 UTC m=+582.969409394"
Jul 17 21:27:04.439456 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:04.439423    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:27:04.443208 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:04.443183    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:27:44.528122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.528089    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"]
Jul 17 21:27:44.530308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.530293    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.532585 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.532565    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 17 21:27:44.532702 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.532626    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 17 21:27:44.533504 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.533483    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-z79wp\""
Jul 17 21:27:44.533626 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.533489    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 17 21:27:44.543631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.543612    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"]
Jul 17 21:27:44.571835 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.571817    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.571928 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.571846    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.572045 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.572027    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2e240-1d31-408f-87af-3b314f848141-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.572082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.572067    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.572118 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.572091    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.572151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.572120    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c42lh\" (UniqueName: \"kubernetes.io/projected/12b2e240-1d31-408f-87af-3b314f848141-kube-api-access-c42lh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672628 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672602    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672628 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672629    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672658    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2e240-1d31-408f-87af-3b314f848141-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672692    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672713    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.672818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.672773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c42lh\" (UniqueName: \"kubernetes.io/projected/12b2e240-1d31-408f-87af-3b314f848141-kube-api-access-c42lh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.673050 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.673029    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.673130 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.673115    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.673186 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.673163    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.675029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.675007    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/12b2e240-1d31-408f-87af-3b314f848141-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.675274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.675258    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/12b2e240-1d31-408f-87af-3b314f848141-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.680153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.680136    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c42lh\" (UniqueName: \"kubernetes.io/projected/12b2e240-1d31-408f-87af-3b314f848141-kube-api-access-c42lh\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5\" (UID: \"12b2e240-1d31-408f-87af-3b314f848141\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.840457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.840376    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:44.973392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.973369    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"]
Jul 17 21:27:44.975662 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:27:44.975360    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12b2e240_1d31_408f_87af_3b314f848141.slice/crio-9eb5ca233cb3e463215bd3ea6fb1f837a095fbcf4b6839c725f88819e5769027 WatchSource:0}: Error finding container 9eb5ca233cb3e463215bd3ea6fb1f837a095fbcf4b6839c725f88819e5769027: Status 404 returned error can't find the container with id 9eb5ca233cb3e463215bd3ea6fb1f837a095fbcf4b6839c725f88819e5769027
Jul 17 21:27:44.977037 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:44.977020    2546 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 17 21:27:45.246521 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:45.246474    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5" event={"ID":"12b2e240-1d31-408f-87af-3b314f848141","Type":"ContainerStarted","Data":"9eb5ca233cb3e463215bd3ea6fb1f837a095fbcf4b6839c725f88819e5769027"}
Jul 17 21:27:46.253225 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:46.253194    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5" event={"ID":"12b2e240-1d31-408f-87af-3b314f848141","Type":"ContainerStarted","Data":"c45297394180fee87c162c02bb04091b040d4aea2d20c1d0a7e5449c488422e1"}
Jul 17 21:27:46.269904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:46.269852    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5" podStartSLOduration=1.0893993339999999 podStartE2EDuration="2.269836044s" totalImagesPullingTime="1.18043671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:27:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:27:46.268246081 +0000 UTC m=+642.248616559" watchObservedRunningTime="2026-07-17 21:27:46.269836044 +0000 UTC m=+642.250206579"
Jul 17 21:27:54.841139 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:54.841088    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:54.841644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:54.841170    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:54.853701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:54.853672    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:27:55.304505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:27:55.304474    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5"
Jul 17 21:28:01.337103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.337059    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"]
Jul 17 21:28:01.344059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.344036    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.346389 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.346364    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 17 21:28:01.352448 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.352426    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"]
Jul 17 21:28:01.406694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406668    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.406798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406696    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.406798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406738    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9ww7\" (UniqueName: \"kubernetes.io/projected/23a6217f-72e7-4b31-9847-d22f68c28332-kube-api-access-v9ww7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.406869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406813    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a6217f-72e7-4b31-9847-d22f68c28332-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.406869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406831    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.406869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.406856    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507568 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507541    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a6217f-72e7-4b31-9847-d22f68c28332-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507570    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507599    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507636    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507658    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.507879 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.507715    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9ww7\" (UniqueName: \"kubernetes.io/projected/23a6217f-72e7-4b31-9847-d22f68c28332-kube-api-access-v9ww7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.508031 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.508008    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.508112 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.508030    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.508216 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.508197    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.509918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.509898    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/23a6217f-72e7-4b31-9847-d22f68c28332-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.510076 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.510060    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/23a6217f-72e7-4b31-9847-d22f68c28332-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.517915 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.517892    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9ww7\" (UniqueName: \"kubernetes.io/projected/23a6217f-72e7-4b31-9847-d22f68c28332-kube-api-access-v9ww7\") pod \"e2e-distinct-simulated-kserve-59975d69f9-k4hr6\" (UID: \"23a6217f-72e7-4b31-9847-d22f68c28332\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.655397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.655343    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:01.787932 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:01.787909    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"]
Jul 17 21:28:01.789735 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:28:01.789700    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23a6217f_72e7_4b31_9847_d22f68c28332.slice/crio-ae30974947982fdb2cd367954557dc2edd80ebad6e871632814194c4004ea446 WatchSource:0}: Error finding container ae30974947982fdb2cd367954557dc2edd80ebad6e871632814194c4004ea446: Status 404 returned error can't find the container with id ae30974947982fdb2cd367954557dc2edd80ebad6e871632814194c4004ea446
Jul 17 21:28:02.327284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:02.327204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6" event={"ID":"23a6217f-72e7-4b31-9847-d22f68c28332","Type":"ContainerStarted","Data":"5d61d1bc7dcdd95f022065a8d23ae21d68351954d8e311188c3bd0978d2975e6"}
Jul 17 21:28:02.327284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:02.327255    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6" event={"ID":"23a6217f-72e7-4b31-9847-d22f68c28332","Type":"ContainerStarted","Data":"ae30974947982fdb2cd367954557dc2edd80ebad6e871632814194c4004ea446"}
Jul 17 21:28:02.347095 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:02.347042    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6" podStartSLOduration=1.081137704 podStartE2EDuration="1.347026261s" totalImagesPullingTime="265.888557ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:28:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:28:02.345006185 +0000 UTC m=+658.325376671" watchObservedRunningTime="2026-07-17 21:28:02.347026261 +0000 UTC m=+658.327396737"
Jul 17 21:28:11.655674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:11.655617    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:11.656179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:11.655710    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:11.668465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:11.668436    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:28:12.389885 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:28:12.389852    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-k4hr6"
Jul 17 21:29:23.416271 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.416184    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:29:23.416798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.416413    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" podUID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" containerName="authorino" containerID="cri-o://9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a" gracePeriod=30
Jul 17 21:29:23.668530 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.668477    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:29:23.670572 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.670555    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcts8\" (UniqueName: \"kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8\") pod \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") "
Jul 17 21:29:23.670618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.670596    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert\") pod \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\" (UID: \"c8a1d39c-8012-42c0-8741-2bbd64c081a6\") "
Jul 17 21:29:23.672667 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.672637    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8" (OuterVolumeSpecName: "kube-api-access-fcts8") pod "c8a1d39c-8012-42c0-8741-2bbd64c081a6" (UID: "c8a1d39c-8012-42c0-8741-2bbd64c081a6"). InnerVolumeSpecName "kube-api-access-fcts8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:29:23.680101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.680077    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "c8a1d39c-8012-42c0-8741-2bbd64c081a6" (UID: "c8a1d39c-8012-42c0-8741-2bbd64c081a6"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 17 21:29:23.708100 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.708071    2546 generic.go:356] "Generic (PLEG): container finished" podID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" containerID="9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a" exitCode=0
Jul 17 21:29:23.708214 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.708115    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-798db8ff7d-pwgjw"
Jul 17 21:29:23.708214 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.708153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" event={"ID":"c8a1d39c-8012-42c0-8741-2bbd64c081a6","Type":"ContainerDied","Data":"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"}
Jul 17 21:29:23.708214 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.708203    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-798db8ff7d-pwgjw" event={"ID":"c8a1d39c-8012-42c0-8741-2bbd64c081a6","Type":"ContainerDied","Data":"240377f41ee6e081b2eccd1a1fdecad048fcdfda48313e5c6c715b5eaf47a3d5"}
Jul 17 21:29:23.708312 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.708220    2546 scope.go:117] "RemoveContainer" containerID="9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"
Jul 17 21:29:23.717854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.717836    2546 scope.go:117] "RemoveContainer" containerID="9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"
Jul 17 21:29:23.718113 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:29:23.718093    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a\": container with ID starting with 9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a not found: ID does not exist" containerID="9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"
Jul 17 21:29:23.718149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.718121    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a"} err="failed to get container status \"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a\": rpc error: code = NotFound desc = could not find container \"9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a\": container with ID starting with 9101e0f078ded254671b99665bf30c092585d044dacb3ab62461f65f3ab06e9a not found: ID does not exist"
Jul 17 21:29:23.732368 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.732345    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:29:23.737628 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.737609    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-798db8ff7d-pwgjw"]
Jul 17 21:29:23.771298 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.771273    2546 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c8a1d39c-8012-42c0-8741-2bbd64c081a6-tls-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:29:23.771298 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:23.771294    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcts8\" (UniqueName: \"kubernetes.io/projected/c8a1d39c-8012-42c0-8741-2bbd64c081a6-kube-api-access-fcts8\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:29:24.536021 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:29:24.535993    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" path="/var/lib/kubelet/pods/c8a1d39c-8012-42c0-8741-2bbd64c081a6/volumes"
Jul 17 21:30:00.140550 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.140514    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"]
Jul 17 21:30:00.141065 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.140817    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" containerName="authorino"
Jul 17 21:30:00.141065 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.140828    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" containerName="authorino"
Jul 17 21:30:00.141065 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.140887    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8a1d39c-8012-42c0-8741-2bbd64c081a6" containerName="authorino"
Jul 17 21:30:00.143403 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.143387    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"
Jul 17 21:30:00.145846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.145816    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:30:00.150896 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.150873    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"]
Jul 17 21:30:00.153661 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.153641    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"
Jul 17 21:30:00.280181 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.280153    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"]
Jul 17 21:30:00.283059 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:30:00.283031    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263b66e2_e7e7_4393_9269_5aad90490125.slice/crio-d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a WatchSource:0}: Error finding container d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a: Status 404 returned error can't find the container with id d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a
Jul 17 21:30:00.874456 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:00.874426    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q" event={"ID":"263b66e2-e7e7-4393-9269-5aad90490125","Type":"ContainerStarted","Data":"d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a"}
Jul 17 21:30:03.891304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:03.891261    2546 generic.go:356] "Generic (PLEG): container finished" podID="263b66e2-e7e7-4393-9269-5aad90490125" containerID="599d82eb3bd883f2b3db4380b35be6faf497a601170435aea0c11a600818429e" exitCode=0
Jul 17 21:30:03.891681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:03.891343    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q" event={"ID":"263b66e2-e7e7-4393-9269-5aad90490125","Type":"ContainerDied","Data":"599d82eb3bd883f2b3db4380b35be6faf497a601170435aea0c11a600818429e"}
Jul 17 21:30:05.023927 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:05.023899    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"
Jul 17 21:30:05.901565 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:05.901537    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"
Jul 17 21:30:05.901827 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:05.901587    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q" event={"ID":"263b66e2-e7e7-4393-9269-5aad90490125","Type":"ContainerDied","Data":"d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a"}
Jul 17 21:30:05.901827 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:30:05.901626    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d288754fc209b70881d4fd36148bc76dfaa15ebd0bdd441abda17cb88b238d5a"
Jul 17 21:32:04.471596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:32:04.471513    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:32:04.472151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:32:04.471744    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:37:04.502149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:37:04.502113    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:37:04.504330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:37:04.504307    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:42:04.530823 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:42:04.530791    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:42:04.534014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:42:04.533993    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:43:33.613344 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.613309    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:43:33.613825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.613616    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="263b66e2-e7e7-4393-9269-5aad90490125" containerName="cleanup"
Jul 17 21:43:33.613825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.613626    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="263b66e2-e7e7-4393-9269-5aad90490125" containerName="cleanup"
Jul 17 21:43:33.613825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.613678    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="263b66e2-e7e7-4393-9269-5aad90490125" containerName="cleanup"
Jul 17 21:43:33.616608 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.616588    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.618905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.618884    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-ddd2a457-openshift-default-dockercfg-pxlcg\""
Jul 17 21:43:33.662132 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662100    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662232 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662136    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662232 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662156    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hg4c\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662232 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662183    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662258    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662303    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662383    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662549 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662425    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.662549 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.662486    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.676356 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.676334    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:43:33.763613 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763580    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.763762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763621    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.763762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763639    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.763762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.763762 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763680    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hg4c\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763916    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763944    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.763987    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.764019    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764294 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.764250    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764294 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.764271    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764402 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.764378    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.764457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.764425    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.766231 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.766213    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.766316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.766301    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.771675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.771646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hg4c\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.771786 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.771767    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token\") pod \"e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") " pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:33.926941 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:33.926872    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:34.057519 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.057498    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:43:34.059968 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:43:34.059943    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod380f66a7_8fdd_496e_8ceb_80625273fb1c.slice/crio-c1b62b873b2e8c1d8700743f2358f53243070c8d7f840b4b904936882cafad17 WatchSource:0}: Error finding container c1b62b873b2e8c1d8700743f2358f53243070c8d7f840b4b904936882cafad17: Status 404 returned error can't find the container with id c1b62b873b2e8c1d8700743f2358f53243070c8d7f840b4b904936882cafad17
Jul 17 21:43:34.062106 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.062071    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:34.062198 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.062181    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:34.062237 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.062226    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:34.685745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.685692    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerStarted","Data":"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"}
Jul 17 21:43:34.686208 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.685765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerStarted","Data":"c1b62b873b2e8c1d8700743f2358f53243070c8d7f840b4b904936882cafad17"}
Jul 17 21:43:34.707922 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.707876    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podStartSLOduration=1.70786216 podStartE2EDuration="1.70786216s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:43:34.70527311 +0000 UTC m=+1590.685643600" watchObservedRunningTime="2026-07-17 21:43:34.70786216 +0000 UTC m=+1590.688232638"
Jul 17 21:43:34.927493 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:34.927457    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:35.931866 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:35.931839    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:36.696199 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:36.696168    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:36.700846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:36.700814    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:39.744805 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.744760    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:43:39.747972 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.747945    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:39.750187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.750169    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:43:39.754829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.754806    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:43:39.814504 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.814472    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x\" (UID: \"dc4108b5-b294-4b6e-bc40-50f87d7de4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:39.914947 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.914911    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x\" (UID: \"dc4108b5-b294-4b6e-bc40-50f87d7de4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:39.915508 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:39.915489    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x\" (UID: \"dc4108b5-b294-4b6e-bc40-50f87d7de4b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:40.060108 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:40.060032    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:40.228154 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:40.228104    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:43:40.230204 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:43:40.230176    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc4108b5_b294_4b6e_bc40_50f87d7de4b2.slice/crio-747b0d5a0beaf1fb42f844065db24813b60619da26a87419918fb6caef503bd4 WatchSource:0}: Error finding container 747b0d5a0beaf1fb42f844065db24813b60619da26a87419918fb6caef503bd4: Status 404 returned error can't find the container with id 747b0d5a0beaf1fb42f844065db24813b60619da26a87419918fb6caef503bd4
Jul 17 21:43:40.714774 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:40.714692    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerStarted","Data":"b4b89f268471e02ebb3570c737120a6ef7a3a29e2eead78db35ba68011bdc4fb"}
Jul 17 21:43:40.714774 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:40.714765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerStarted","Data":"747b0d5a0beaf1fb42f844065db24813b60619da26a87419918fb6caef503bd4"}
Jul 17 21:43:40.733065 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:40.733020    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" podStartSLOduration=1.733005693 podStartE2EDuration="1.733005693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:43:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:43:40.730491596 +0000 UTC m=+1596.710862073" watchObservedRunningTime="2026-07-17 21:43:40.733005693 +0000 UTC m=+1596.713376212"
Jul 17 21:43:41.720493 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:41.720455    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerID="b4b89f268471e02ebb3570c737120a6ef7a3a29e2eead78db35ba68011bdc4fb" exitCode=7
Jul 17 21:43:41.720905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:41.720526    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerDied","Data":"b4b89f268471e02ebb3570c737120a6ef7a3a29e2eead78db35ba68011bdc4fb"}
Jul 17 21:43:41.720905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:41.720790    2546 scope.go:117] "RemoveContainer" containerID="b4b89f268471e02ebb3570c737120a6ef7a3a29e2eead78db35ba68011bdc4fb"
Jul 17 21:43:42.727848 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:42.727769    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerID="c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7" exitCode=7
Jul 17 21:43:42.728162 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:42.727852    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerDied","Data":"c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7"}
Jul 17 21:43:42.728162 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:42.727916    2546 scope.go:117] "RemoveContainer" containerID="b4b89f268471e02ebb3570c737120a6ef7a3a29e2eead78db35ba68011bdc4fb"
Jul 17 21:43:42.728162 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:42.727992    2546 scope.go:117] "RemoveContainer" containerID="c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7"
Jul 17 21:43:42.728265 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:43:42.728220    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x_odh-ai-gateway-infra(dc4108b5-b294-4b6e-bc40-50f87d7de4b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2"
Jul 17 21:43:43.735315 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:43.735289    2546 scope.go:117] "RemoveContainer" containerID="c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7"
Jul 17 21:43:43.735693 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:43:43.735478    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x_odh-ai-gateway-infra(dc4108b5-b294-4b6e-bc40-50f87d7de4b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2"
Jul 17 21:43:46.750677 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.750642    2546 generic.go:356] "Generic (PLEG): container finished" podID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerID="04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13" exitCode=0
Jul 17 21:43:46.751163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.750741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerDied","Data":"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"}
Jul 17 21:43:46.751163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.751089    2546 scope.go:117] "RemoveContainer" containerID="04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"
Jul 17 21:43:46.752086 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.752062    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:46.752164 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.752133    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:46.752201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:46.752169    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:43:47.757407 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:47.757378    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerStarted","Data":"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"}
Jul 17 21:43:47.927453 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:47.927419    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:47.928366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:47.928346    2546 patch_prober.go:28] interesting pod/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 17 21:43:47.928423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:47.928393    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 17 21:43:48.927533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:48.927498    2546 patch_prober.go:28] interesting pod/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 17 21:43:48.927923 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:48.927547    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 17 21:43:49.927854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:49.927824    2546 patch_prober.go:28] interesting pod/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 17 21:43:49.928218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:49.927872    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 17 21:43:50.927690 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:50.927656    2546 patch_prober.go:28] interesting pod/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused" start-of-body=
Jul 17 21:43:50.927899 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:50.927710    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.38:15021/healthz/ready\": dial tcp 10.133.0.38:15021: connect: connection refused"
Jul 17 21:43:51.696369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:51.696336    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:51.930957 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:51.930930    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:51.932101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:51.932082    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:43:55.531312 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:55.531245    2546 scope.go:117] "RemoveContainer" containerID="c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7"
Jul 17 21:43:55.804207 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:55.804154    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerStarted","Data":"5e962baaf1a506510afee593d861159797bda9015ea14aaee3da546e5c32a8be"}
Jul 17 21:43:56.809960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:56.809931    2546 generic.go:356] "Generic (PLEG): container finished" podID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerID="5e962baaf1a506510afee593d861159797bda9015ea14aaee3da546e5c32a8be" exitCode=7
Jul 17 21:43:56.810354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:56.809977    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerDied","Data":"5e962baaf1a506510afee593d861159797bda9015ea14aaee3da546e5c32a8be"}
Jul 17 21:43:56.810354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:56.810021    2546 scope.go:117] "RemoveContainer" containerID="c49a9a3fd034df094c75981c9ba11e1c74d29443a8bae7cd1f7b144644a150a7"
Jul 17 21:43:56.810354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:56.810235    2546 scope.go:117] "RemoveContainer" containerID="5e962baaf1a506510afee593d861159797bda9015ea14aaee3da546e5c32a8be"
Jul 17 21:43:56.810477 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:43:56.810394    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x_odh-ai-gateway-infra(dc4108b5-b294-4b6e-bc40-50f87d7de4b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2"
Jul 17 21:43:56.835129 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:56.835107    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:43:57.939825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:57.939804    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:58.059236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.059210    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca\") pod \"dc4108b5-b294-4b6e-bc40-50f87d7de4b2\" (UID: \"dc4108b5-b294-4b6e-bc40-50f87d7de4b2\") "
Jul 17 21:43:58.059518 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.059496    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc4108b5-b294-4b6e-bc40-50f87d7de4b2" (UID: "dc4108b5-b294-4b6e-bc40-50f87d7de4b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:43:58.159818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.159753    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc4108b5-b294-4b6e-bc40-50f87d7de4b2-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:43:58.820996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.820921    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"
Jul 17 21:43:58.820996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.820965    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x" event={"ID":"dc4108b5-b294-4b6e-bc40-50f87d7de4b2","Type":"ContainerDied","Data":"747b0d5a0beaf1fb42f844065db24813b60619da26a87419918fb6caef503bd4"}
Jul 17 21:43:58.821194 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.821027    2546 scope.go:117] "RemoveContainer" containerID="5e962baaf1a506510afee593d861159797bda9015ea14aaee3da546e5c32a8be"
Jul 17 21:43:58.839641 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.839611    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:43:58.846308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:43:58.846288    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-psn4x"]
Jul 17 21:44:00.535663 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:00.535623    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" path="/var/lib/kubelet/pods/dc4108b5-b294-4b6e-bc40-50f87d7de4b2/volumes"
Jul 17 21:44:29.799336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799305    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799584    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799594    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799618    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799626    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799668    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799677    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.799750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.799684    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:44:29.803788 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.803773    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:29.807153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.807132    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:44:29.814604 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.814570    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:29.981303 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:29.981265    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-djw69\" (UID: \"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:30.081602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.081539    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-djw69\" (UID: \"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:30.082122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.082105    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-djw69\" (UID: \"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:30.113700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.113681    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:30.239705 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.239679    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:30.241611 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:44:30.241587    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f319d6d_f0b3_42e6_b0fa_0434fe13aa4e.slice/crio-48c8b59b78ad7ed5ec67de455a7edb333af489ad322352ea2142ce144d218291 WatchSource:0}: Error finding container 48c8b59b78ad7ed5ec67de455a7edb333af489ad322352ea2142ce144d218291: Status 404 returned error can't find the container with id 48c8b59b78ad7ed5ec67de455a7edb333af489ad322352ea2142ce144d218291
Jul 17 21:44:30.962523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.962496    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerID="bab25ffe4cdcebb259fa27806fff4b4294d95c5f2a3e82c28307e2592106c0d2" exitCode=7
Jul 17 21:44:30.962893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.962580    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerDied","Data":"bab25ffe4cdcebb259fa27806fff4b4294d95c5f2a3e82c28307e2592106c0d2"}
Jul 17 21:44:30.962893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.962643    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerStarted","Data":"48c8b59b78ad7ed5ec67de455a7edb333af489ad322352ea2142ce144d218291"}
Jul 17 21:44:30.962893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:30.962750    2546 scope.go:117] "RemoveContainer" containerID="bab25ffe4cdcebb259fa27806fff4b4294d95c5f2a3e82c28307e2592106c0d2"
Jul 17 21:44:31.969476 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:31.969406    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerID="76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba" exitCode=7
Jul 17 21:44:31.969861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:31.969465    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerDied","Data":"76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba"}
Jul 17 21:44:31.969861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:31.969525    2546 scope.go:117] "RemoveContainer" containerID="bab25ffe4cdcebb259fa27806fff4b4294d95c5f2a3e82c28307e2592106c0d2"
Jul 17 21:44:31.969861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:31.969664    2546 scope.go:117] "RemoveContainer" containerID="76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba"
Jul 17 21:44:31.969977 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:44:31.969876    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-djw69_odh-ai-gateway-infra(3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e"
Jul 17 21:44:32.975549 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:32.975519    2546 scope.go:117] "RemoveContainer" containerID="76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba"
Jul 17 21:44:32.975954 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:44:32.975711    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-djw69_odh-ai-gateway-infra(3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e"
Jul 17 21:44:46.531398 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:46.531361    2546 scope.go:117] "RemoveContainer" containerID="76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba"
Jul 17 21:44:47.043170 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:47.043134    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerStarted","Data":"b7f74546d79de4f1a26362d04367c25fcf8f6dc741a56bedd56d933f5606d9f9"}
Jul 17 21:44:47.062487 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:47.062436    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" podStartSLOduration=18.062420427 podStartE2EDuration="18.062420427s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:44:47.060793384 +0000 UTC m=+1663.041163862" watchObservedRunningTime="2026-07-17 21:44:47.062420427 +0000 UTC m=+1663.042790937"
Jul 17 21:44:48.050142 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:48.050111    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerID="b7f74546d79de4f1a26362d04367c25fcf8f6dc741a56bedd56d933f5606d9f9" exitCode=7
Jul 17 21:44:48.050142 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:48.050145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerDied","Data":"b7f74546d79de4f1a26362d04367c25fcf8f6dc741a56bedd56d933f5606d9f9"}
Jul 17 21:44:48.050640 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:48.050193    2546 scope.go:117] "RemoveContainer" containerID="76ca420cc6355a7fdc68505aa2aa2bbce19ae7f45fe7e753160ec7ae501b3fba"
Jul 17 21:44:48.050640 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:48.050444    2546 scope.go:117] "RemoveContainer" containerID="b7f74546d79de4f1a26362d04367c25fcf8f6dc741a56bedd56d933f5606d9f9"
Jul 17 21:44:48.050766 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:44:48.050650    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-djw69_odh-ai-gateway-infra(3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e"
Jul 17 21:44:48.077433 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:48.077410    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:49.182376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:49.182353    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:49.237297 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:49.237268    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca\") pod \"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e\" (UID: \"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e\") "
Jul 17 21:44:49.237627 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:49.237605    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" (UID: "3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:44:49.337995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:49.337924    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:44:50.062200 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.062176    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"
Jul 17 21:44:50.062399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.062174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69" event={"ID":"3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e","Type":"ContainerDied","Data":"48c8b59b78ad7ed5ec67de455a7edb333af489ad322352ea2142ce144d218291"}
Jul 17 21:44:50.062399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.062314    2546 scope.go:117] "RemoveContainer" containerID="b7f74546d79de4f1a26362d04367c25fcf8f6dc741a56bedd56d933f5606d9f9"
Jul 17 21:44:50.083700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.083676    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:50.091935 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.091915    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-djw69"]
Jul 17 21:44:50.535836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:44:50.535808    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" path="/var/lib/kubelet/pods/3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e/volumes"
Jul 17 21:45:00.143356 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143324    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143675    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143688    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143703    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143708    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc4108b5-b294-4b6e-bc40-50f87d7de4b2" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143716    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143745    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143807    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.143855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.143816    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.147953 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.147935    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"]
Jul 17 21:45:00.148067 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.148047    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"
Jul 17 21:45:00.148273 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.148257    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.148310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.148276    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.148373 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.148361    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f319d6d-f0b3-42e6-b0fa-0434fe13aa4e" containerName="revoke-keys"
Jul 17 21:45:00.150652 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.150632    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:45:00.150991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.150976    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"
Jul 17 21:45:00.155417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.155389    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"]
Jul 17 21:45:00.158229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.158204    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:00.160753 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.160713    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"
Jul 17 21:45:00.167651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.167632    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"
Jul 17 21:45:00.304893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.304862    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"]
Jul 17 21:45:00.305486 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:45:00.305444    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda21fc198_48e6_4e00_96c7_c1a9257af5cf.slice/crio-29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620 WatchSource:0}: Error finding container 29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620: Status 404 returned error can't find the container with id 29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620
Jul 17 21:45:00.318912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:00.318884    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:00.320365 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:45:00.320342    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0796453_fff2_4a49_8012_06092f373ae3.slice/crio-8ef74477fe78c1fa55a88f4c12e3f78978b08c7f6aa583d572867b55d0bfe572 WatchSource:0}: Error finding container 8ef74477fe78c1fa55a88f4c12e3f78978b08c7f6aa583d572867b55d0bfe572: Status 404 returned error can't find the container with id 8ef74477fe78c1fa55a88f4c12e3f78978b08c7f6aa583d572867b55d0bfe572
Jul 17 21:45:01.120514 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.120472    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745" event={"ID":"a21fc198-48e6-4e00-96c7-c1a9257af5cf","Type":"ContainerStarted","Data":"a01fcfce7524030f18e4c59d9892ef868ed03dbc7af3dcb57b15049197b137c3"}
Jul 17 21:45:01.120659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.120540    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745" event={"ID":"a21fc198-48e6-4e00-96c7-c1a9257af5cf","Type":"ContainerStarted","Data":"29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620"}
Jul 17 21:45:01.122022 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.122000    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0796453-fff2-4a49-8012-06092f373ae3" containerID="2382eb20aecddacd3ef1528e5fc85422f921e13ac4af8e15abbaab5e9ece9f9b" exitCode=7
Jul 17 21:45:01.122100 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.122033    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerDied","Data":"2382eb20aecddacd3ef1528e5fc85422f921e13ac4af8e15abbaab5e9ece9f9b"}
Jul 17 21:45:01.122100 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.122065    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerStarted","Data":"8ef74477fe78c1fa55a88f4c12e3f78978b08c7f6aa583d572867b55d0bfe572"}
Jul 17 21:45:01.122188 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.122174    2546 scope.go:117] "RemoveContainer" containerID="2382eb20aecddacd3ef1528e5fc85422f921e13ac4af8e15abbaab5e9ece9f9b"
Jul 17 21:45:01.137421 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:01.137379    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745" podStartSLOduration=1.137366877 podStartE2EDuration="1.137366877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:01.135431777 +0000 UTC m=+1677.115802253" watchObservedRunningTime="2026-07-17 21:45:01.137366877 +0000 UTC m=+1677.117737362"
Jul 17 21:45:02.129010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.128917    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0796453-fff2-4a49-8012-06092f373ae3" containerID="54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2" exitCode=7
Jul 17 21:45:02.129448 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.129015    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerDied","Data":"54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2"}
Jul 17 21:45:02.129448 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.129081    2546 scope.go:117] "RemoveContainer" containerID="2382eb20aecddacd3ef1528e5fc85422f921e13ac4af8e15abbaab5e9ece9f9b"
Jul 17 21:45:02.129448 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.129170    2546 scope.go:117] "RemoveContainer" containerID="54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2"
Jul 17 21:45:02.129448 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:02.129409    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d_odh-ai-gateway-infra(d0796453-fff2-4a49-8012-06092f373ae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" podUID="d0796453-fff2-4a49-8012-06092f373ae3"
Jul 17 21:45:02.130759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.130739    2546 generic.go:356] "Generic (PLEG): container finished" podID="a21fc198-48e6-4e00-96c7-c1a9257af5cf" containerID="a01fcfce7524030f18e4c59d9892ef868ed03dbc7af3dcb57b15049197b137c3" exitCode=0
Jul 17 21:45:02.130840 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:02.130765    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745" event={"ID":"a21fc198-48e6-4e00-96c7-c1a9257af5cf","Type":"ContainerDied","Data":"a01fcfce7524030f18e4c59d9892ef868ed03dbc7af3dcb57b15049197b137c3"}
Jul 17 21:45:03.137432 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:03.137389    2546 scope.go:117] "RemoveContainer" containerID="54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2"
Jul 17 21:45:03.137837 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:03.137644    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d_odh-ai-gateway-infra(d0796453-fff2-4a49-8012-06092f373ae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" podUID="d0796453-fff2-4a49-8012-06092f373ae3"
Jul 17 21:45:03.268548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:03.268523    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"
Jul 17 21:45:04.142424 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:04.142394    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745"
Jul 17 21:45:04.142930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:04.142432    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738745-sl745" event={"ID":"a21fc198-48e6-4e00-96c7-c1a9257af5cf","Type":"ContainerDied","Data":"29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620"}
Jul 17 21:45:04.142930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:04.142473    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29a179d750b153353bdf96fe0a6a8fb98b3c8d6697ae3e3e5b862e38edb48620"
Jul 17 21:45:15.530922 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:15.530893    2546 scope.go:117] "RemoveContainer" containerID="54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2"
Jul 17 21:45:16.198446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:16.198353    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerStarted","Data":"1661668c710402f1445afd6ea6741cfab9008b31f655bf5177d6cc456ab39484"}
Jul 17 21:45:16.214475 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:16.214415    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" podStartSLOduration=16.214396843 podStartE2EDuration="16.214396843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:45:16.213413989 +0000 UTC m=+1692.193784465" watchObservedRunningTime="2026-07-17 21:45:16.214396843 +0000 UTC m=+1692.194767322"
Jul 17 21:45:17.205296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:17.205262    2546 generic.go:356] "Generic (PLEG): container finished" podID="d0796453-fff2-4a49-8012-06092f373ae3" containerID="1661668c710402f1445afd6ea6741cfab9008b31f655bf5177d6cc456ab39484" exitCode=7
Jul 17 21:45:17.205757 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:17.205334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerDied","Data":"1661668c710402f1445afd6ea6741cfab9008b31f655bf5177d6cc456ab39484"}
Jul 17 21:45:17.205757 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:17.205396    2546 scope.go:117] "RemoveContainer" containerID="54d76cc9f4579f2f0debbbe2e718f15aa55a3961e4453617f4e69f356cb696e2"
Jul 17 21:45:17.205757 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:17.205600    2546 scope.go:117] "RemoveContainer" containerID="1661668c710402f1445afd6ea6741cfab9008b31f655bf5177d6cc456ab39484"
Jul 17 21:45:17.205870 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:17.205840    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d_odh-ai-gateway-infra(d0796453-fff2-4a49-8012-06092f373ae3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" podUID="d0796453-fff2-4a49-8012-06092f373ae3"
Jul 17 21:45:17.233511 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:17.233479    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:18.346998 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:18.346971    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"
Jul 17 21:45:19.218278 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:19.218251    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"
Jul 17 21:45:19.218278 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:19.218257    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d" event={"ID":"d0796453-fff2-4a49-8012-06092f373ae3","Type":"ContainerDied","Data":"8ef74477fe78c1fa55a88f4c12e3f78978b08c7f6aa583d572867b55d0bfe572"}
Jul 17 21:45:19.218506 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:19.218338    2546 scope.go:117] "RemoveContainer" containerID="1661668c710402f1445afd6ea6741cfab9008b31f655bf5177d6cc456ab39484"
Jul 17 21:45:19.235551 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:19.235529    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:19.241097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:19.241069    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ait-ddd2a457-29738745-fr88d"]
Jul 17 21:45:20.535836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:20.535804    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0796453-fff2-4a49-8012-06092f373ae3" path="/var/lib/kubelet/pods/d0796453-fff2-4a49-8012-06092f373ae3/volumes"
Jul 17 21:45:29.863036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863003    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863306    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a21fc198-48e6-4e00-96c7-c1a9257af5cf" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863317    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a21fc198-48e6-4e00-96c7-c1a9257af5cf" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863334    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863340    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863347    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863352    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863360    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863365    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863414    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863421    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863427    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0796453-fff2-4a49-8012-06092f373ae3" containerName="cleanup"
Jul 17 21:45:29.863454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.863436    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a21fc198-48e6-4e00-96c7-c1a9257af5cf" containerName="cleanup"
Jul 17 21:45:29.867707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.867686    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:29.870654 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.870631    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:45:29.878205 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.878181    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:29.951046 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:29.951017    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5\" (UID: \"df0bb4c9-3232-4959-b0e5-b2dcceccc90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:30.051924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:30.051891    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5\" (UID: \"df0bb4c9-3232-4959-b0e5-b2dcceccc90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:30.052480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:30.052460    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5\" (UID: \"df0bb4c9-3232-4959-b0e5-b2dcceccc90c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:30.180080 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:30.180021    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:30.307739 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:30.307702    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:30.309789 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:45:30.309763    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0bb4c9_3232_4959_b0e5_b2dcceccc90c.slice/crio-9ececfa5550eadec41e8cf447194cd77913d0512607556cff49004c3a018d396 WatchSource:0}: Error finding container 9ececfa5550eadec41e8cf447194cd77913d0512607556cff49004c3a018d396: Status 404 returned error can't find the container with id 9ececfa5550eadec41e8cf447194cd77913d0512607556cff49004c3a018d396
Jul 17 21:45:31.282289 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:31.282252    2546 generic.go:356] "Generic (PLEG): container finished" podID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerID="1eaab7faf0dfecf178df0bb88644c75f0f4d77a920a776bf9132eac0a9ade7ef" exitCode=7
Jul 17 21:45:31.282797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:31.282342    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" event={"ID":"df0bb4c9-3232-4959-b0e5-b2dcceccc90c","Type":"ContainerDied","Data":"1eaab7faf0dfecf178df0bb88644c75f0f4d77a920a776bf9132eac0a9ade7ef"}
Jul 17 21:45:31.282797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:31.282398    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" event={"ID":"df0bb4c9-3232-4959-b0e5-b2dcceccc90c","Type":"ContainerStarted","Data":"9ececfa5550eadec41e8cf447194cd77913d0512607556cff49004c3a018d396"}
Jul 17 21:45:31.282797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:31.282482    2546 scope.go:117] "RemoveContainer" containerID="1eaab7faf0dfecf178df0bb88644c75f0f4d77a920a776bf9132eac0a9ade7ef"
Jul 17 21:45:32.289140 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:32.289107    2546 generic.go:356] "Generic (PLEG): container finished" podID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerID="61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2" exitCode=7
Jul 17 21:45:32.289542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:32.289165    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" event={"ID":"df0bb4c9-3232-4959-b0e5-b2dcceccc90c","Type":"ContainerDied","Data":"61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2"}
Jul 17 21:45:32.289542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:32.289240    2546 scope.go:117] "RemoveContainer" containerID="1eaab7faf0dfecf178df0bb88644c75f0f4d77a920a776bf9132eac0a9ade7ef"
Jul 17 21:45:32.289542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:32.289379    2546 scope.go:117] "RemoveContainer" containerID="61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2"
Jul 17 21:45:32.289697 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:32.289597    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5_odh-ai-gateway-infra(df0bb4c9-3232-4959-b0e5-b2dcceccc90c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c"
Jul 17 21:45:33.295672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:33.295644    2546 scope.go:117] "RemoveContainer" containerID="61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2"
Jul 17 21:45:33.296072 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:33.295888    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5_odh-ai-gateway-infra(df0bb4c9-3232-4959-b0e5-b2dcceccc90c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c"
Jul 17 21:45:48.531247 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:48.531211    2546 scope.go:117] "RemoveContainer" containerID="61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2"
Jul 17 21:45:49.379612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:49.379570    2546 generic.go:356] "Generic (PLEG): container finished" podID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerID="d9e773c831a7cb98581d96546d0b313f2027f70faa64e307a47d832137ce13f5" exitCode=7
Jul 17 21:45:49.379752 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:49.379647    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" event={"ID":"df0bb4c9-3232-4959-b0e5-b2dcceccc90c","Type":"ContainerDied","Data":"d9e773c831a7cb98581d96546d0b313f2027f70faa64e307a47d832137ce13f5"}
Jul 17 21:45:49.379752 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:49.379711    2546 scope.go:117] "RemoveContainer" containerID="61f64d30eb3447ba830f19617d37d16642817f4a401a381fc9b81decda4ff2c2"
Jul 17 21:45:49.379997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:49.379978    2546 scope.go:117] "RemoveContainer" containerID="d9e773c831a7cb98581d96546d0b313f2027f70faa64e307a47d832137ce13f5"
Jul 17 21:45:49.380163 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:45:49.380141    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5_odh-ai-gateway-infra(df0bb4c9-3232-4959-b0e5-b2dcceccc90c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c"
Jul 17 21:45:50.408828 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:50.408789    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:50.541299 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:50.541274    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:50.703187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:50.703093    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca\") pod \"df0bb4c9-3232-4959-b0e5-b2dcceccc90c\" (UID: \"df0bb4c9-3232-4959-b0e5-b2dcceccc90c\") "
Jul 17 21:45:50.703470 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:50.703445    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "df0bb4c9-3232-4959-b0e5-b2dcceccc90c" (UID: "df0bb4c9-3232-4959-b0e5-b2dcceccc90c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:45:50.804520 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:50.804492    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/df0bb4c9-3232-4959-b0e5-b2dcceccc90c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:45:51.392838 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:51.392806    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"
Jul 17 21:45:51.392838 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:51.392825    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5" event={"ID":"df0bb4c9-3232-4959-b0e5-b2dcceccc90c","Type":"ContainerDied","Data":"9ececfa5550eadec41e8cf447194cd77913d0512607556cff49004c3a018d396"}
Jul 17 21:45:51.393104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:51.392882    2546 scope.go:117] "RemoveContainer" containerID="d9e773c831a7cb98581d96546d0b313f2027f70faa64e307a47d832137ce13f5"
Jul 17 21:45:51.413708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:51.413686    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:51.421873 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:51.421853    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xwvp5"]
Jul 17 21:45:52.536202 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:45:52.536164    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" path="/var/lib/kubelet/pods/df0bb4c9-3232-4959-b0e5-b2dcceccc90c/volumes"
Jul 17 21:46:29.909240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909204    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909504    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909515    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909525    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909530    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909592    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.909688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.909602    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:29.912385 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.912368    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:29.914808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.914782    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:46:29.921147 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.921126    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:29.993537 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:29.993514    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc\" (UID: \"04ec52cf-881b-4c94-aba6-cf144a5eeef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:30.094289 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:30.094263    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc\" (UID: \"04ec52cf-881b-4c94-aba6-cf144a5eeef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:30.094863 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:30.094844    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc\" (UID: \"04ec52cf-881b-4c94-aba6-cf144a5eeef0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:30.224165 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:30.224108    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:30.355442 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:30.355418    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:30.357546 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:46:30.357523    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04ec52cf_881b_4c94_aba6_cf144a5eeef0.slice/crio-ff39f3b22ca8aa5025b777eb00ea9d0b72a1da6d018a15fb1534dbbf16731812 WatchSource:0}: Error finding container ff39f3b22ca8aa5025b777eb00ea9d0b72a1da6d018a15fb1534dbbf16731812: Status 404 returned error can't find the container with id ff39f3b22ca8aa5025b777eb00ea9d0b72a1da6d018a15fb1534dbbf16731812
Jul 17 21:46:30.590439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:30.590407    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" event={"ID":"04ec52cf-881b-4c94-aba6-cf144a5eeef0","Type":"ContainerStarted","Data":"ff39f3b22ca8aa5025b777eb00ea9d0b72a1da6d018a15fb1534dbbf16731812"}
Jul 17 21:46:31.596401 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:31.596363    2546 generic.go:356] "Generic (PLEG): container finished" podID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerID="fb4a56a5e13b38b39aa3d5ebfbd8ef3d2071e8d537dff1d08430f90e1f753226" exitCode=7
Jul 17 21:46:31.596883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:31.596448    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" event={"ID":"04ec52cf-881b-4c94-aba6-cf144a5eeef0","Type":"ContainerDied","Data":"fb4a56a5e13b38b39aa3d5ebfbd8ef3d2071e8d537dff1d08430f90e1f753226"}
Jul 17 21:46:31.596883 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:31.596593    2546 scope.go:117] "RemoveContainer" containerID="fb4a56a5e13b38b39aa3d5ebfbd8ef3d2071e8d537dff1d08430f90e1f753226"
Jul 17 21:46:32.602819 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:32.602787    2546 generic.go:356] "Generic (PLEG): container finished" podID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerID="8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23" exitCode=7
Jul 17 21:46:32.603194 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:32.602823    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" event={"ID":"04ec52cf-881b-4c94-aba6-cf144a5eeef0","Type":"ContainerDied","Data":"8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23"}
Jul 17 21:46:32.603194 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:32.602872    2546 scope.go:117] "RemoveContainer" containerID="fb4a56a5e13b38b39aa3d5ebfbd8ef3d2071e8d537dff1d08430f90e1f753226"
Jul 17 21:46:32.603194 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:32.603109    2546 scope.go:117] "RemoveContainer" containerID="8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23"
Jul 17 21:46:32.603342 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:46:32.603303    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc_odh-ai-gateway-infra(04ec52cf-881b-4c94-aba6-cf144a5eeef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0"
Jul 17 21:46:33.610960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:33.610927    2546 scope.go:117] "RemoveContainer" containerID="8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23"
Jul 17 21:46:33.611407 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:46:33.611155    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc_odh-ai-gateway-infra(04ec52cf-881b-4c94-aba6-cf144a5eeef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0"
Jul 17 21:46:39.871416 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:39.871381    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:46:39.871995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:39.871629    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy" containerID="cri-o://e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007" gracePeriod=30
Jul 17 21:46:45.121518 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.121496    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:46:45.209636 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209610    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.209773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209652    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.209773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209690    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.209773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209740    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.209933 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209773    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.209933 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209822    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.210035 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.209926    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.210088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210047    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9hg4c\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.210141 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210089    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy\") pod \"380f66a7-8fdd-496e-8ceb-80625273fb1c\" (UID: \"380f66a7-8fdd-496e-8ceb-80625273fb1c\") "
Jul 17 21:46:45.210456 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210330    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:45.211310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210555    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:45.211310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210566    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data" (OuterVolumeSpecName: "istio-data") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:45.211310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.210821    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:45.211310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.211119    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:45.214068 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.214032    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:46:45.215796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.215735    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c" (OuterVolumeSpecName: "kube-api-access-9hg4c") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "kube-api-access-9hg4c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:46:45.218057 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.218029    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token" (OuterVolumeSpecName: "istio-token") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:46:45.218397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.218378    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "380f66a7-8fdd-496e-8ceb-80625273fb1c" (UID: "380f66a7-8fdd-496e-8ceb-80625273fb1c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:46:45.311668 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311643    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311668 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311670    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/380f66a7-8fdd-496e-8ceb-80625273fb1c-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311682    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hg4c\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-kube-api-access-9hg4c\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311691    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311699    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311706    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311713    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311732    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/380f66a7-8fdd-496e-8ceb-80625273fb1c-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.311841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.311743    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/380f66a7-8fdd-496e-8ceb-80625273fb1c-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:45.672154 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.672116    2546 generic.go:356] "Generic (PLEG): container finished" podID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerID="e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007" exitCode=0
Jul 17 21:46:45.672534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.672153    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerDied","Data":"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"}
Jul 17 21:46:45.672534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.672182    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"
Jul 17 21:46:45.672534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.672208    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq" event={"ID":"380f66a7-8fdd-496e-8ceb-80625273fb1c","Type":"ContainerDied","Data":"c1b62b873b2e8c1d8700743f2358f53243070c8d7f840b4b904936882cafad17"}
Jul 17 21:46:45.672534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.672232    2546 scope.go:117] "RemoveContainer" containerID="e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"
Jul 17 21:46:45.682921 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.682908    2546 scope.go:117] "RemoveContainer" containerID="04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"
Jul 17 21:46:45.690981 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.690943    2546 scope.go:117] "RemoveContainer" containerID="e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"
Jul 17 21:46:45.691371 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:46:45.691306    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007\": container with ID starting with e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007 not found: ID does not exist" containerID="e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"
Jul 17 21:46:45.691371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.691342    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007"} err="failed to get container status \"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007\": rpc error: code = NotFound desc = could not find container \"e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007\": container with ID starting with e1afe0d5683ce6fe014b9d3fb0a47a3f3ff4f691de5f6d98cf2c86ed05f07007 not found: ID does not exist"
Jul 17 21:46:45.691371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.691364    2546 scope.go:117] "RemoveContainer" containerID="04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"
Jul 17 21:46:45.691708 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:46:45.691679    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13\": container with ID starting with 04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13 not found: ID does not exist" containerID="04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"
Jul 17 21:46:45.691708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.691701    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13"} err="failed to get container status \"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13\": rpc error: code = NotFound desc = could not find container \"04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13\": container with ID starting with 04fe00a9a21bebb6b30778c0d8166b0479a056f6324949b719328525922aba13 not found: ID does not exist"
Jul 17 21:46:45.693215 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.693197    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:46:45.698801 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:45.698781    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-ddd2a457-openshift-default-69fdf576f-mpfpq"]
Jul 17 21:46:46.537231 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:46.537196    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" path="/var/lib/kubelet/pods/380f66a7-8fdd-496e-8ceb-80625273fb1c/volumes"
Jul 17 21:46:48.531637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:48.531594    2546 scope.go:117] "RemoveContainer" containerID="8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23"
Jul 17 21:46:49.699367 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:49.699335    2546 generic.go:356] "Generic (PLEG): container finished" podID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerID="d3936e11e7b1e7f146d61bc799e7cfb2e0b5aad8c738544f32b0c18ed8d34765" exitCode=6
Jul 17 21:46:49.699884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:49.699413    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" event={"ID":"04ec52cf-881b-4c94-aba6-cf144a5eeef0","Type":"ContainerDied","Data":"d3936e11e7b1e7f146d61bc799e7cfb2e0b5aad8c738544f32b0c18ed8d34765"}
Jul 17 21:46:49.699884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:49.699534    2546 scope.go:117] "RemoveContainer" containerID="8e4d8c6ef4a0bf92337c7585fb6c622a9ac32d1efcc27df4574a83fdfb7f1f23"
Jul 17 21:46:49.699884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:49.699872    2546 scope.go:117] "RemoveContainer" containerID="d3936e11e7b1e7f146d61bc799e7cfb2e0b5aad8c738544f32b0c18ed8d34765"
Jul 17 21:46:49.700079 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:46:49.700058    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc_odh-ai-gateway-infra(04ec52cf-881b-4c94-aba6-cf144a5eeef0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0"
Jul 17 21:46:50.729562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:50.729524    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:50.861082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:50.861058    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:50.955077 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:50.955045    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca\") pod \"04ec52cf-881b-4c94-aba6-cf144a5eeef0\" (UID: \"04ec52cf-881b-4c94-aba6-cf144a5eeef0\") "
Jul 17 21:46:50.955406 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:50.955386    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "04ec52cf-881b-4c94-aba6-cf144a5eeef0" (UID: "04ec52cf-881b-4c94-aba6-cf144a5eeef0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:46:51.056439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.056394    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/04ec52cf-881b-4c94-aba6-cf144a5eeef0-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:46:51.711515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.711487    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"
Jul 17 21:46:51.711515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.711499    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc" event={"ID":"04ec52cf-881b-4c94-aba6-cf144a5eeef0","Type":"ContainerDied","Data":"ff39f3b22ca8aa5025b777eb00ea9d0b72a1da6d018a15fb1534dbbf16731812"}
Jul 17 21:46:51.711770 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.711577    2546 scope.go:117] "RemoveContainer" containerID="d3936e11e7b1e7f146d61bc799e7cfb2e0b5aad8c738544f32b0c18ed8d34765"
Jul 17 21:46:51.731650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.731622    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:51.736138 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:51.736117    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-rrvtc"]
Jul 17 21:46:52.315455 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315425    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:46:52.315749 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315715    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315749 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315748    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315757    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315762    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315768    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315773    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315780    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315785    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315792    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315796    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315851    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315858    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="df0bb4c9-3232-4959-b0e5-b2dcceccc90c" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315864    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315870    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:52.315925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.315877    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:52.320311 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.320293    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.322886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.322868    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-da1f2be7-gw-openshift-default-dockercfg-b9pqg\""
Jul 17 21:46:52.334453 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.334428    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:46:52.466701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466677    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466845 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466707    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466845 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466753    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466845 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466810    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466974 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466852    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466974 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466883    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466974 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466913    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vzkz\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.466974 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466968    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.467114 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.466985    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.535532 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.535505    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" path="/var/lib/kubelet/pods/04ec52cf-881b-4c94-aba6-cf144a5eeef0/volumes"
Jul 17 21:46:52.567830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567773    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vzkz\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.567830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567803    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.567830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567824    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567881    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567901    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567932    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.567972    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568264 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568333 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568319    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568419 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568403    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568474 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568428    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568483    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.568896 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.568876    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.570312 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.570293    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.570535 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.570517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.582566 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.582539    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.582760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.582739    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vzkz\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz\") pod \"e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") " pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.631348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.631328    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:52.774705 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.774683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:46:52.776838 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:46:52.776818    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6147b5cd_e54f_4fb3_8b91_8c88e2a88df2.slice/crio-72b03bb76e950083a05b1f3cd721171ac2b3b5aad63482acd975ef6cf2d0358e WatchSource:0}: Error finding container 72b03bb76e950083a05b1f3cd721171ac2b3b5aad63482acd975ef6cf2d0358e: Status 404 returned error can't find the container with id 72b03bb76e950083a05b1f3cd721171ac2b3b5aad63482acd975ef6cf2d0358e
Jul 17 21:46:52.778824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.778793    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:46:52.778913 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.778851    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:46:52.778913 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:52.778878    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:46:53.725706 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:53.725666    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerStarted","Data":"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"}
Jul 17 21:46:53.725899 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:53.725742    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerStarted","Data":"72b03bb76e950083a05b1f3cd721171ac2b3b5aad63482acd975ef6cf2d0358e"}
Jul 17 21:46:53.747857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:53.747808    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podStartSLOduration=1.747795365 podStartE2EDuration="1.747795365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:46:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:46:53.745760617 +0000 UTC m=+1789.726131093" watchObservedRunningTime="2026-07-17 21:46:53.747795365 +0000 UTC m=+1789.728165842"
Jul 17 21:46:54.631571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:54.631542    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:54.636458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:54.636432    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:54.730480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:54.730457    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:54.731166 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:54.731146    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:46:58.087369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.087340    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:46:58.087753 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.087632    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:58.087753 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.087644    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="380f66a7-8fdd-496e-8ceb-80625273fb1c" containerName="istio-proxy"
Jul 17 21:46:58.087753 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.087702    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="04ec52cf-881b-4c94-aba6-cf144a5eeef0" containerName="revoke-keys"
Jul 17 21:46:58.090654 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.090636    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:46:58.092687 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.092662    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:46:58.097681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.097658    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:46:58.211405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.211375    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw\" (UID: \"35756471-64b6-44e5-80c9-3e106b944472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:46:58.312634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.312608    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw\" (UID: \"35756471-64b6-44e5-80c9-3e106b944472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:46:58.313177 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.313161    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw\" (UID: \"35756471-64b6-44e5-80c9-3e106b944472\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:46:58.403015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.402966    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:46:58.530228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.530205    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:46:58.532361 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:46:58.532325    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35756471_64b6_44e5_80c9_3e106b944472.slice/crio-0cf49718400a830f7d5f1f3fd7a9f86849aabb63b373f8881ea1ddf7fde54679 WatchSource:0}: Error finding container 0cf49718400a830f7d5f1f3fd7a9f86849aabb63b373f8881ea1ddf7fde54679: Status 404 returned error can't find the container with id 0cf49718400a830f7d5f1f3fd7a9f86849aabb63b373f8881ea1ddf7fde54679
Jul 17 21:46:58.748549 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:58.748517    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" event={"ID":"35756471-64b6-44e5-80c9-3e106b944472","Type":"ContainerStarted","Data":"0cf49718400a830f7d5f1f3fd7a9f86849aabb63b373f8881ea1ddf7fde54679"}
Jul 17 21:46:59.754152 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:59.754118    2546 generic.go:356] "Generic (PLEG): container finished" podID="35756471-64b6-44e5-80c9-3e106b944472" containerID="f00f5846f782e0b949632bf871b122612451346881a65b09aa1d51281ba1cb8a" exitCode=7
Jul 17 21:46:59.754647 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:59.754199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" event={"ID":"35756471-64b6-44e5-80c9-3e106b944472","Type":"ContainerDied","Data":"f00f5846f782e0b949632bf871b122612451346881a65b09aa1d51281ba1cb8a"}
Jul 17 21:46:59.754647 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:46:59.754374    2546 scope.go:117] "RemoveContainer" containerID="f00f5846f782e0b949632bf871b122612451346881a65b09aa1d51281ba1cb8a"
Jul 17 21:47:00.761575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:00.761541    2546 generic.go:356] "Generic (PLEG): container finished" podID="35756471-64b6-44e5-80c9-3e106b944472" containerID="72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d" exitCode=7
Jul 17 21:47:00.762034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:00.761627    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" event={"ID":"35756471-64b6-44e5-80c9-3e106b944472","Type":"ContainerDied","Data":"72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d"}
Jul 17 21:47:00.762034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:00.761684    2546 scope.go:117] "RemoveContainer" containerID="f00f5846f782e0b949632bf871b122612451346881a65b09aa1d51281ba1cb8a"
Jul 17 21:47:00.762034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:00.761872    2546 scope.go:117] "RemoveContainer" containerID="72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d"
Jul 17 21:47:00.762167 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:00.762065    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw_odh-ai-gateway-infra(35756471-64b6-44e5-80c9-3e106b944472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" podUID="35756471-64b6-44e5-80c9-3e106b944472"
Jul 17 21:47:01.768401 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:01.768373    2546 scope.go:117] "RemoveContainer" containerID="72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d"
Jul 17 21:47:01.768827 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:01.768539    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw_odh-ai-gateway-infra(35756471-64b6-44e5-80c9-3e106b944472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" podUID="35756471-64b6-44e5-80c9-3e106b944472"
Jul 17 21:47:04.585611 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:04.585574    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:47:04.589871 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:04.589852    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:47:09.732093 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.731994    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 21:47:09.732093 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.732076    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 21:47:09.816251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.816224    2546 generic.go:356] "Generic (PLEG): container finished" podID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerID="9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae" exitCode=0
Jul 17 21:47:09.816400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.816306    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerDied","Data":"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"}
Jul 17 21:47:09.816668 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.816652    2546 scope.go:117] "RemoveContainer" containerID="9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"
Jul 17 21:47:09.817590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.817566    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:47:09.817643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.817629    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:47:09.817678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:09.817658    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:47:10.829280 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:10.829247    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerStarted","Data":"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"}
Jul 17 21:47:11.632436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:11.632402    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:47:11.632625 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:11.632598    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 21:47:11.632670 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:11.632644    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 21:47:12.531887 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:12.531856    2546 scope.go:117] "RemoveContainer" containerID="72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d"
Jul 17 21:47:12.632652 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:12.632621    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 21:47:12.632810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:12.632685    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 21:47:13.632166 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.632125    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 21:47:13.632616 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.632188    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 21:47:13.846355 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.846323    2546 generic.go:356] "Generic (PLEG): container finished" podID="35756471-64b6-44e5-80c9-3e106b944472" containerID="4eced922e658310604b4bf4a9fc0a173485b9230d28d9083c81b0312a026a96f" exitCode=7
Jul 17 21:47:13.846515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.846399    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" event={"ID":"35756471-64b6-44e5-80c9-3e106b944472","Type":"ContainerDied","Data":"4eced922e658310604b4bf4a9fc0a173485b9230d28d9083c81b0312a026a96f"}
Jul 17 21:47:13.846515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.846455    2546 scope.go:117] "RemoveContainer" containerID="72d0720a704c15735880ec95a01e1ae8c93d0ba98e590009e5a31d109fe8319d"
Jul 17 21:47:13.846813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:13.846791    2546 scope.go:117] "RemoveContainer" containerID="4eced922e658310604b4bf4a9fc0a173485b9230d28d9083c81b0312a026a96f"
Jul 17 21:47:13.847033 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:13.847009    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw_odh-ai-gateway-infra(35756471-64b6-44e5-80c9-3e106b944472)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" podUID="35756471-64b6-44e5-80c9-3e106b944472"
Jul 17 21:47:14.632526 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:14.632490    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body=
Jul 17 21:47:14.632951 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:14.632541    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused"
Jul 17 21:47:14.872411 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:14.872378    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:47:15.004055 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.004032    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:47:15.053218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.053187    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca\") pod \"35756471-64b6-44e5-80c9-3e106b944472\" (UID: \"35756471-64b6-44e5-80c9-3e106b944472\") "
Jul 17 21:47:15.053504 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.053483    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "35756471-64b6-44e5-80c9-3e106b944472" (UID: "35756471-64b6-44e5-80c9-3e106b944472"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:15.154132 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.154106    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/35756471-64b6-44e5-80c9-3e106b944472-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:47:15.635689 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.635653    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:47:15.636115 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.635985    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:47:15.636662 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.636645    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:47:15.858957 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.858930    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"
Jul 17 21:47:15.858957 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.858941    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw" event={"ID":"35756471-64b6-44e5-80c9-3e106b944472","Type":"ContainerDied","Data":"0cf49718400a830f7d5f1f3fd7a9f86849aabb63b373f8881ea1ddf7fde54679"}
Jul 17 21:47:15.859153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.859008    2546 scope.go:117] "RemoveContainer" containerID="4eced922e658310604b4bf4a9fc0a173485b9230d28d9083c81b0312a026a96f"
Jul 17 21:47:15.879876 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.879848    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:47:15.886371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:15.886320    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-4b8pw"]
Jul 17 21:47:16.536929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:16.536897    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35756471-64b6-44e5-80c9-3e106b944472" path="/var/lib/kubelet/pods/35756471-64b6-44e5-80c9-3e106b944472/volumes"
Jul 17 21:47:29.957162 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957122    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957436    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957448    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957465    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957470    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957483    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957488    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957535    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957542    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.957583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.957550    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="35756471-64b6-44e5-80c9-3e106b944472" containerName="revoke-keys"
Jul 17 21:47:29.960111 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.960093    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:29.962673 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.962650    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:47:29.970521 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:29.970486    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:30.073347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.073302    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w765j\" (UID: \"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:30.174273 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.174232    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w765j\" (UID: \"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:30.174984 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.174958    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w765j\" (UID: \"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:30.272376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.272338    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:30.413202 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.413175    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:30.415185 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:47:30.415154    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd2a7be2_8cc2_4582_b20b_3c66201e9f1c.slice/crio-d2ce7488aa0dfe645e292cae3b9bf4aa2d91ece35c62a71502a4447ce3397290 WatchSource:0}: Error finding container d2ce7488aa0dfe645e292cae3b9bf4aa2d91ece35c62a71502a4447ce3397290: Status 404 returned error can't find the container with id d2ce7488aa0dfe645e292cae3b9bf4aa2d91ece35c62a71502a4447ce3397290
Jul 17 21:47:30.936802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.936697    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerStarted","Data":"a0a7ea05ff4eb61dbf725d313b5fca3e970e10121224e14d9028fa0269fc9a99"}
Jul 17 21:47:30.936802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.936784    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerStarted","Data":"d2ce7488aa0dfe645e292cae3b9bf4aa2d91ece35c62a71502a4447ce3397290"}
Jul 17 21:47:30.954483 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:30.954438    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" podStartSLOduration=1.954423877 podStartE2EDuration="1.954423877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:30.952242324 +0000 UTC m=+1826.932612802" watchObservedRunningTime="2026-07-17 21:47:30.954423877 +0000 UTC m=+1826.934794353"
Jul 17 21:47:31.943686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:31.943651    2546 generic.go:356] "Generic (PLEG): container finished" podID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerID="a0a7ea05ff4eb61dbf725d313b5fca3e970e10121224e14d9028fa0269fc9a99" exitCode=6
Jul 17 21:47:31.944134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:31.943772    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerDied","Data":"a0a7ea05ff4eb61dbf725d313b5fca3e970e10121224e14d9028fa0269fc9a99"}
Jul 17 21:47:31.944134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:31.943918    2546 scope.go:117] "RemoveContainer" containerID="a0a7ea05ff4eb61dbf725d313b5fca3e970e10121224e14d9028fa0269fc9a99"
Jul 17 21:47:32.950617 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:32.950528    2546 generic.go:356] "Generic (PLEG): container finished" podID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerID="e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1" exitCode=6
Jul 17 21:47:32.951074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:32.950625    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerDied","Data":"e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1"}
Jul 17 21:47:32.951074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:32.950680    2546 scope.go:117] "RemoveContainer" containerID="a0a7ea05ff4eb61dbf725d313b5fca3e970e10121224e14d9028fa0269fc9a99"
Jul 17 21:47:32.951074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:32.950847    2546 scope.go:117] "RemoveContainer" containerID="e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1"
Jul 17 21:47:32.951189 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:32.951072    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w765j_odh-ai-gateway-infra(fd2a7be2-8cc2-4582-b20b-3c66201e9f1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c"
Jul 17 21:47:33.956661 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:33.956629    2546 scope.go:117] "RemoveContainer" containerID="e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1"
Jul 17 21:47:33.957065 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:33.956858    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w765j_odh-ai-gateway-infra(fd2a7be2-8cc2-4582-b20b-3c66201e9f1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c"
Jul 17 21:47:48.136316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.136277    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:47:48.138645 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.138627    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:47:48.146170 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.146137    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:47:48.229634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.229593    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h\" (UID: \"8caad3f4-0910-42bb-8185-6db159630d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:47:48.330447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.330404    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h\" (UID: \"8caad3f4-0910-42bb-8185-6db159630d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:47:48.331073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.331054    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h\" (UID: \"8caad3f4-0910-42bb-8185-6db159630d32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:47:48.450464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.450381    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:47:48.530936 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.530895    2546 scope.go:117] "RemoveContainer" containerID="e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1"
Jul 17 21:47:48.598159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:48.598129    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:47:48.599902 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:47:48.599863    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8caad3f4_0910_42bb_8185_6db159630d32.slice/crio-19b00cee044178498714f5f92e719ca79a8870e961408d9d869317d41717e14d WatchSource:0}: Error finding container 19b00cee044178498714f5f92e719ca79a8870e961408d9d869317d41717e14d: Status 404 returned error can't find the container with id 19b00cee044178498714f5f92e719ca79a8870e961408d9d869317d41717e14d
Jul 17 21:47:49.033979 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:49.033888    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerStarted","Data":"741a39aa4658b3244e75b95b599f7aff1c840e443f7cb53d8cc4c0d26f5d8d09"}
Jul 17 21:47:49.035288 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:49.035264    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerStarted","Data":"0e90c95a8a39230e69634997b251c9e68054a3461cf17716e71505b6e9f74cca"}
Jul 17 21:47:49.035442 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:49.035302    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerStarted","Data":"19b00cee044178498714f5f92e719ca79a8870e961408d9d869317d41717e14d"}
Jul 17 21:47:49.069983 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:49.069922    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" podStartSLOduration=1.069904249 podStartE2EDuration="1.069904249s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:47:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:47:49.068023451 +0000 UTC m=+1845.048393929" watchObservedRunningTime="2026-07-17 21:47:49.069904249 +0000 UTC m=+1845.050274725"
Jul 17 21:47:50.042418 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.042382    2546 generic.go:356] "Generic (PLEG): container finished" podID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerID="741a39aa4658b3244e75b95b599f7aff1c840e443f7cb53d8cc4c0d26f5d8d09" exitCode=6
Jul 17 21:47:50.042895 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.042453    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerDied","Data":"741a39aa4658b3244e75b95b599f7aff1c840e443f7cb53d8cc4c0d26f5d8d09"}
Jul 17 21:47:50.042895 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.042524    2546 scope.go:117] "RemoveContainer" containerID="e6e004c84f7d3141ff7694295ee85192e87a3ae33740bbcc1e751611284101a1"
Jul 17 21:47:50.042895 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.042883    2546 scope.go:117] "RemoveContainer" containerID="741a39aa4658b3244e75b95b599f7aff1c840e443f7cb53d8cc4c0d26f5d8d09"
Jul 17 21:47:50.043134 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:50.043105    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w765j_odh-ai-gateway-infra(fd2a7be2-8cc2-4582-b20b-3c66201e9f1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c"
Jul 17 21:47:50.044184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.044159    2546 generic.go:356] "Generic (PLEG): container finished" podID="8caad3f4-0910-42bb-8185-6db159630d32" containerID="0e90c95a8a39230e69634997b251c9e68054a3461cf17716e71505b6e9f74cca" exitCode=7
Jul 17 21:47:50.044294 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.044237    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerDied","Data":"0e90c95a8a39230e69634997b251c9e68054a3461cf17716e71505b6e9f74cca"}
Jul 17 21:47:50.044413 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.044399    2546 scope.go:117] "RemoveContainer" containerID="0e90c95a8a39230e69634997b251c9e68054a3461cf17716e71505b6e9f74cca"
Jul 17 21:47:50.067656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:50.067634    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:51.052453 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.052426    2546 generic.go:356] "Generic (PLEG): container finished" podID="8caad3f4-0910-42bb-8185-6db159630d32" containerID="a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57" exitCode=7
Jul 17 21:47:51.052802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.052508    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerDied","Data":"a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57"}
Jul 17 21:47:51.052802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.052572    2546 scope.go:117] "RemoveContainer" containerID="0e90c95a8a39230e69634997b251c9e68054a3461cf17716e71505b6e9f74cca"
Jul 17 21:47:51.052802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.052715    2546 scope.go:117] "RemoveContainer" containerID="a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57"
Jul 17 21:47:51.052962 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:51.052943    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h_odh-ai-gateway-infra(8caad3f4-0910-42bb-8185-6db159630d32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" podUID="8caad3f4-0910-42bb-8185-6db159630d32"
Jul 17 21:47:51.175230 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.175207    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:51.251241 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.251216    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca\") pod \"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c\" (UID: \"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c\") "
Jul 17 21:47:51.251566 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.251542    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" (UID: "fd2a7be2-8cc2-4582-b20b-3c66201e9f1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:47:51.352084 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:51.352010    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:47:52.058860 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.058831    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"
Jul 17 21:47:52.059318 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.058826    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j" event={"ID":"fd2a7be2-8cc2-4582-b20b-3c66201e9f1c","Type":"ContainerDied","Data":"d2ce7488aa0dfe645e292cae3b9bf4aa2d91ece35c62a71502a4447ce3397290"}
Jul 17 21:47:52.059318 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.058985    2546 scope.go:117] "RemoveContainer" containerID="741a39aa4658b3244e75b95b599f7aff1c840e443f7cb53d8cc4c0d26f5d8d09"
Jul 17 21:47:52.061117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.061092    2546 scope.go:117] "RemoveContainer" containerID="a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57"
Jul 17 21:47:52.061368 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:47:52.061343    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h_odh-ai-gateway-infra(8caad3f4-0910-42bb-8185-6db159630d32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" podUID="8caad3f4-0910-42bb-8185-6db159630d32"
Jul 17 21:47:52.095620 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.095590    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:52.101319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.101299    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w765j"]
Jul 17 21:47:52.536078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:52.536049    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" path="/var/lib/kubelet/pods/fd2a7be2-8cc2-4582-b20b-3c66201e9f1c/volumes"
Jul 17 21:47:58.201122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:58.201071    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:47:58.201579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:47:58.201332    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" containerID="cri-o://a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3" gracePeriod=30
Jul 17 21:48:00.638872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:00.638834    2546 patch_prober.go:28] interesting pod/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 21:48:00.639376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:00.638894    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 21:48:02.531317 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:02.531286    2546 scope.go:117] "RemoveContainer" containerID="a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57"
Jul 17 21:48:03.118294 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.118263    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerStarted","Data":"060bc34b64c5de993f8eb220cf2efbc34ed66267c0c6e277e7cff5a1697a0c97"}
Jul 17 21:48:03.450523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.450497    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:48:03.548338 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548313    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548362    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548380    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548413    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548438    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548477    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548501    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vzkz\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548528    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548566    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data\") pod \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\" (UID: \"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2\") "
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548687    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:48:03.548802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548739    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:48:03.549464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548916    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.549464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.548935    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.549464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.549140    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:48:03.549464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.549219    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:03.549884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.549858    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data" (OuterVolumeSpecName: "istio-data") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:48:03.551084 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.551052    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token" (OuterVolumeSpecName: "istio-token") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:48:03.551204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.551119    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz" (OuterVolumeSpecName: "kube-api-access-9vzkz") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "kube-api-access-9vzkz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:48:03.551204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.551143    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:48:03.551407 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.551388    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" (UID: "6147b5cd-e54f-4fb3-8b91-8c88e2a88df2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:48:03.650139 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650098    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650139 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650138    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vzkz\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-kube-api-access-9vzkz\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650152    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650166    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650179    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650190    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:03.650295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:03.650202    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:04.125396 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.125359    2546 generic.go:356] "Generic (PLEG): container finished" podID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerID="a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3" exitCode=0
Jul 17 21:48:04.125570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.125407    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"
Jul 17 21:48:04.125570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.125454    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerDied","Data":"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"}
Jul 17 21:48:04.125570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.125514    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr" event={"ID":"6147b5cd-e54f-4fb3-8b91-8c88e2a88df2","Type":"ContainerDied","Data":"72b03bb76e950083a05b1f3cd721171ac2b3b5aad63482acd975ef6cf2d0358e"}
Jul 17 21:48:04.125570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.125551    2546 scope.go:117] "RemoveContainer" containerID="a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"
Jul 17 21:48:04.127346 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.127325    2546 generic.go:356] "Generic (PLEG): container finished" podID="8caad3f4-0910-42bb-8185-6db159630d32" containerID="060bc34b64c5de993f8eb220cf2efbc34ed66267c0c6e277e7cff5a1697a0c97" exitCode=7
Jul 17 21:48:04.127426 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.127404    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerDied","Data":"060bc34b64c5de993f8eb220cf2efbc34ed66267c0c6e277e7cff5a1697a0c97"}
Jul 17 21:48:04.127766 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.127749    2546 scope.go:117] "RemoveContainer" containerID="060bc34b64c5de993f8eb220cf2efbc34ed66267c0c6e277e7cff5a1697a0c97"
Jul 17 21:48:04.127991 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:04.127971    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h_odh-ai-gateway-infra(8caad3f4-0910-42bb-8185-6db159630d32)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" podUID="8caad3f4-0910-42bb-8185-6db159630d32"
Jul 17 21:48:04.135512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.135497    2546 scope.go:117] "RemoveContainer" containerID="9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"
Jul 17 21:48:04.143314 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.143257    2546 scope.go:117] "RemoveContainer" containerID="a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"
Jul 17 21:48:04.143566 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:04.143534    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3\": container with ID starting with a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3 not found: ID does not exist" containerID="a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"
Jul 17 21:48:04.143669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.143565    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3"} err="failed to get container status \"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3\": rpc error: code = NotFound desc = could not find container \"a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3\": container with ID starting with a765e42af9604069b93c9c1554b67b222bb82cc8bf8ae2671b9164768114bbf3 not found: ID does not exist"
Jul 17 21:48:04.143669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.143587    2546 scope.go:117] "RemoveContainer" containerID="9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"
Jul 17 21:48:04.143991 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:04.143973    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae\": container with ID starting with 9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae not found: ID does not exist" containerID="9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"
Jul 17 21:48:04.144067 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.144000    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae"} err="failed to get container status \"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae\": rpc error: code = NotFound desc = could not find container \"9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae\": container with ID starting with 9ca434dfbaac839610ded5200ae643b038c10e5aa2363a214c9dcc7b74eb3dae not found: ID does not exist"
Jul 17 21:48:04.144067 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.144021    2546 scope.go:117] "RemoveContainer" containerID="a9e7f63843d52f99c4e31aed0e617a0a0c28038e6ee18e794e84f63027034b57"
Jul 17 21:48:04.145784 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.145767    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:48:04.172135 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.172112    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:48:04.177712 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.177693    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-da1f2be7-gw-openshift-default-58d856bb5f-2h8tr"]
Jul 17 21:48:04.540570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:04.540539    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" path="/var/lib/kubelet/pods/6147b5cd-e54f-4fb3-8b91-8c88e2a88df2/volumes"
Jul 17 21:48:05.269116 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:05.269088    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:48:05.362824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:05.362780    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca\") pod \"8caad3f4-0910-42bb-8185-6db159630d32\" (UID: \"8caad3f4-0910-42bb-8185-6db159630d32\") "
Jul 17 21:48:05.363226 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:05.363199    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8caad3f4-0910-42bb-8185-6db159630d32" (UID: "8caad3f4-0910-42bb-8185-6db159630d32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:05.463768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:05.463646    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8caad3f4-0910-42bb-8185-6db159630d32-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:06.139320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.139293    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"
Jul 17 21:48:06.139607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.139339    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h" event={"ID":"8caad3f4-0910-42bb-8185-6db159630d32","Type":"ContainerDied","Data":"19b00cee044178498714f5f92e719ca79a8870e961408d9d869317d41717e14d"}
Jul 17 21:48:06.139607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.139411    2546 scope.go:117] "RemoveContainer" containerID="060bc34b64c5de993f8eb220cf2efbc34ed66267c0c6e277e7cff5a1697a0c97"
Jul 17 21:48:06.160595 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.160572    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:48:06.166467 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.166446    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b8g5h"]
Jul 17 21:48:06.536850 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:06.536823    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8caad3f4-0910-42bb-8185-6db159630d32" path="/var/lib/kubelet/pods/8caad3f4-0910-42bb-8185-6db159630d32/volumes"
Jul 17 21:48:10.953005 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.952964    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:48:10.953485 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953430    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953485 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953446    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953485 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953471    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953485 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953480    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953489    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953497    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953512    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953520    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953535    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953543    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953553    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953561    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953578    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953585    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953661    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953673    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953682    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6147b5cd-e54f-4fb3-8b91-8c88e2a88df2" containerName="istio-proxy"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953691    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953700    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:10.953700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.953709    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:10.956757 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.956710    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:10.959309 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.959285    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-b3d03bc3-openshift-default-dockercfg-9hvq4\""
Jul 17 21:48:10.971929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:10.971893    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:48:11.106059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106021    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106057    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106076    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106098    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prk2g\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106181    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106236    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106295    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106341    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.106580 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.106381    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207442    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207509    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207530    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207548    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207572    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prk2g\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207782    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207847    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207907    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.207987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.207964    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.208254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.208147    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.208334 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.208314    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.208417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.208304    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.208497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.208368    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.208538 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.208384    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.210027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.210004    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.210309 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.210289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.221093 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.221050    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.229435 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.229407    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prk2g\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g\") pod \"e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") " pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.267367 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.267319    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:11.415353 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.415255    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:48:11.417486 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:48:11.417460    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b4c5214_824a_4027_9646_aa9e951f04e8.slice/crio-8206859b69ca6f137fa326d05e67545d37745eafc1dd8ac914c62f0bb85a3ead WatchSource:0}: Error finding container 8206859b69ca6f137fa326d05e67545d37745eafc1dd8ac914c62f0bb85a3ead: Status 404 returned error can't find the container with id 8206859b69ca6f137fa326d05e67545d37745eafc1dd8ac914c62f0bb85a3ead
Jul 17 21:48:11.419517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.419484    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:11.419590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.419553    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:11.419638 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:11.419588    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:12.171609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:12.171560    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerStarted","Data":"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"}
Jul 17 21:48:12.172050 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:12.171638    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerStarted","Data":"8206859b69ca6f137fa326d05e67545d37745eafc1dd8ac914c62f0bb85a3ead"}
Jul 17 21:48:12.207610 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:12.207550    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podStartSLOduration=2.207535176 podStartE2EDuration="2.207535176s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:12.206093939 +0000 UTC m=+1868.186464432" watchObservedRunningTime="2026-07-17 21:48:12.207535176 +0000 UTC m=+1868.187905654"
Jul 17 21:48:12.267467 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:12.267427    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:12.272992 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:12.272962    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:13.176894 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:13.176864    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:13.177981 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:13.177962    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:17.223942 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.223901    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:17.224440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.224329    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:17.224440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.224347    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:17.224440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.224429    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8caad3f4-0910-42bb-8185-6db159630d32" containerName="revoke-keys"
Jul 17 21:48:17.224440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.224443    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd2a7be2-8cc2-4582-b20b-3c66201e9f1c" containerName="revoke-keys"
Jul 17 21:48:17.227573 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.227552    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:17.232007 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.231987    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:48:17.255074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.255040    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:17.265439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.265407    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd\" (UID: \"795f1089-f704-4827-a178-a9d53330c397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:17.366209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.366178    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd\" (UID: \"795f1089-f704-4827-a178-a9d53330c397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:17.367176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.367152    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd\" (UID: \"795f1089-f704-4827-a178-a9d53330c397\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:17.538236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.538207    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:17.720568 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:17.720546    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:17.722470 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:48:17.722441    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod795f1089_f704_4827_a178_a9d53330c397.slice/crio-5d98063a777cde79f33a8778bccf616a94f3e673316c3c0a859489cf3227721c WatchSource:0}: Error finding container 5d98063a777cde79f33a8778bccf616a94f3e673316c3c0a859489cf3227721c: Status 404 returned error can't find the container with id 5d98063a777cde79f33a8778bccf616a94f3e673316c3c0a859489cf3227721c
Jul 17 21:48:18.204650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:18.204615    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerStarted","Data":"68a0898f112b2a3b5a4ebd43b3684c47a8165079241634843d9b7e661757751b"}
Jul 17 21:48:18.204812 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:18.204673    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerStarted","Data":"5d98063a777cde79f33a8778bccf616a94f3e673316c3c0a859489cf3227721c"}
Jul 17 21:48:18.221017 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:18.220973    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" podStartSLOduration=1.2209578859999999 podStartE2EDuration="1.220957886s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:18.219510567 +0000 UTC m=+1874.199881043" watchObservedRunningTime="2026-07-17 21:48:18.220957886 +0000 UTC m=+1874.201328441"
Jul 17 21:48:19.210521 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:19.210483    2546 generic.go:356] "Generic (PLEG): container finished" podID="795f1089-f704-4827-a178-a9d53330c397" containerID="68a0898f112b2a3b5a4ebd43b3684c47a8165079241634843d9b7e661757751b" exitCode=7
Jul 17 21:48:19.210928 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:19.210521    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerDied","Data":"68a0898f112b2a3b5a4ebd43b3684c47a8165079241634843d9b7e661757751b"}
Jul 17 21:48:19.210928 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:19.210790    2546 scope.go:117] "RemoveContainer" containerID="68a0898f112b2a3b5a4ebd43b3684c47a8165079241634843d9b7e661757751b"
Jul 17 21:48:20.218000 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:20.217965    2546 generic.go:356] "Generic (PLEG): container finished" podID="795f1089-f704-4827-a178-a9d53330c397" containerID="95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3" exitCode=7
Jul 17 21:48:20.218441 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:20.218056    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerDied","Data":"95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3"}
Jul 17 21:48:20.218441 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:20.218136    2546 scope.go:117] "RemoveContainer" containerID="68a0898f112b2a3b5a4ebd43b3684c47a8165079241634843d9b7e661757751b"
Jul 17 21:48:20.218441 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:20.218344    2546 scope.go:117] "RemoveContainer" containerID="95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3"
Jul 17 21:48:20.218581 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:20.218560    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd_odh-ai-gateway-infra(795f1089-f704-4827-a178-a9d53330c397)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" podUID="795f1089-f704-4827-a178-a9d53330c397"
Jul 17 21:48:21.226106 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:21.226080    2546 scope.go:117] "RemoveContainer" containerID="95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3"
Jul 17 21:48:21.226469 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:21.226317    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd_odh-ai-gateway-infra(795f1089-f704-4827-a178-a9d53330c397)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" podUID="795f1089-f704-4827-a178-a9d53330c397"
Jul 17 21:48:23.237608 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.237576    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerID="0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46" exitCode=0
Jul 17 21:48:23.238104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.237654    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerDied","Data":"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"}
Jul 17 21:48:23.238104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.238031    2546 scope.go:117] "RemoveContainer" containerID="0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"
Jul 17 21:48:23.238979 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.238956    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:23.239039 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.239019    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:23.239074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:23.239053    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:48:24.244946 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:24.244911    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerStarted","Data":"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"}
Jul 17 21:48:24.267500 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:24.267465    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:24.268954 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:24.268633    2546 patch_prober.go:28] interesting pod/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:48:24.268954 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:24.268687    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:48:25.268014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:25.267966    2546 patch_prober.go:28] interesting pod/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:48:25.268535 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:25.268038    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:48:26.267911 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:26.267876    2546 patch_prober.go:28] interesting pod/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:48:26.268203 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:26.268171    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:48:27.267763 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:27.267697    2546 patch_prober.go:28] interesting pod/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body=
Jul 17 21:48:27.267977 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:27.267800    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused"
Jul 17 21:48:28.177444 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:28.177399    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:28.271375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:28.271347    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:28.272376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:28.272354    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:48:30.012438 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.012399    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:30.019899 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.019869    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:30.023858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.023808    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:30.069836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.069797    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s\" (UID: \"e5c417a9-856a-4c3a-b90b-a911eee90ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:30.170376 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.170322    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s\" (UID: \"e5c417a9-856a-4c3a-b90b-a911eee90ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:30.170990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.170972    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s\" (UID: \"e5c417a9-856a-4c3a-b90b-a911eee90ecf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:30.334410 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.334317    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:30.468922 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:30.468897    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:30.471120 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:48:30.471091    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c417a9_856a_4c3a_b90b_a911eee90ecf.slice/crio-2dade5a8eb22970b1fa16e07977c95d63f781fee070f0ba9348ff85c2defdb95 WatchSource:0}: Error finding container 2dade5a8eb22970b1fa16e07977c95d63f781fee070f0ba9348ff85c2defdb95: Status 404 returned error can't find the container with id 2dade5a8eb22970b1fa16e07977c95d63f781fee070f0ba9348ff85c2defdb95
Jul 17 21:48:31.280897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:31.280868    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerID="e1f71b8d41326a512f99e16aba009c9b63de910c621bcfdee3ee01462f32f436" exitCode=6
Jul 17 21:48:31.281206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:31.280911    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" event={"ID":"e5c417a9-856a-4c3a-b90b-a911eee90ecf","Type":"ContainerDied","Data":"e1f71b8d41326a512f99e16aba009c9b63de910c621bcfdee3ee01462f32f436"}
Jul 17 21:48:31.281206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:31.280952    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" event={"ID":"e5c417a9-856a-4c3a-b90b-a911eee90ecf","Type":"ContainerStarted","Data":"2dade5a8eb22970b1fa16e07977c95d63f781fee070f0ba9348ff85c2defdb95"}
Jul 17 21:48:31.281206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:31.281097    2546 scope.go:117] "RemoveContainer" containerID="e1f71b8d41326a512f99e16aba009c9b63de910c621bcfdee3ee01462f32f436"
Jul 17 21:48:32.288012 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:32.287969    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerID="6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1" exitCode=6
Jul 17 21:48:32.288491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:32.288056    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" event={"ID":"e5c417a9-856a-4c3a-b90b-a911eee90ecf","Type":"ContainerDied","Data":"6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1"}
Jul 17 21:48:32.288491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:32.288133    2546 scope.go:117] "RemoveContainer" containerID="e1f71b8d41326a512f99e16aba009c9b63de910c621bcfdee3ee01462f32f436"
Jul 17 21:48:32.288491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:32.288391    2546 scope.go:117] "RemoveContainer" containerID="6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1"
Jul 17 21:48:32.288653 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:32.288630    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s_odh-ai-gateway-infra(e5c417a9-856a-4c3a-b90b-a911eee90ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf"
Jul 17 21:48:33.294978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:33.294952    2546 scope.go:117] "RemoveContainer" containerID="6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1"
Jul 17 21:48:33.295382 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:33.295122    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s_odh-ai-gateway-infra(e5c417a9-856a-4c3a-b90b-a911eee90ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf"
Jul 17 21:48:34.534094 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:34.534056    2546 scope.go:117] "RemoveContainer" containerID="95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3"
Jul 17 21:48:35.311185 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:35.307207    2546 generic.go:356] "Generic (PLEG): container finished" podID="795f1089-f704-4827-a178-a9d53330c397" containerID="e998ec59c16b207a69e29d8a1769024367ae11229365bb1b33967ee3890779bb" exitCode=7
Jul 17 21:48:35.311185 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:35.307333    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerDied","Data":"e998ec59c16b207a69e29d8a1769024367ae11229365bb1b33967ee3890779bb"}
Jul 17 21:48:35.311185 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:35.307442    2546 scope.go:117] "RemoveContainer" containerID="95c3906ef799fe267b6507f01d6ea36faa9bd167e38ff31402ae0a67fd12c9a3"
Jul 17 21:48:35.311185 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:35.307889    2546 scope.go:117] "RemoveContainer" containerID="e998ec59c16b207a69e29d8a1769024367ae11229365bb1b33967ee3890779bb"
Jul 17 21:48:35.311185 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:35.308838    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd_odh-ai-gateway-infra(795f1089-f704-4827-a178-a9d53330c397)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" podUID="795f1089-f704-4827-a178-a9d53330c397"
Jul 17 21:48:36.334794 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:36.334755    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:36.468077 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:36.468052    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:36.517097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:36.517063    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca\") pod \"795f1089-f704-4827-a178-a9d53330c397\" (UID: \"795f1089-f704-4827-a178-a9d53330c397\") "
Jul 17 21:48:36.517419 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:36.517396    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "795f1089-f704-4827-a178-a9d53330c397" (UID: "795f1089-f704-4827-a178-a9d53330c397"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:36.618406 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:36.618308    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/795f1089-f704-4827-a178-a9d53330c397-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:37.319391 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:37.319359    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd" event={"ID":"795f1089-f704-4827-a178-a9d53330c397","Type":"ContainerDied","Data":"5d98063a777cde79f33a8778bccf616a94f3e673316c3c0a859489cf3227721c"}
Jul 17 21:48:37.319391 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:37.319394    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"
Jul 17 21:48:37.319615 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:37.319419    2546 scope.go:117] "RemoveContainer" containerID="e998ec59c16b207a69e29d8a1769024367ae11229365bb1b33967ee3890779bb"
Jul 17 21:48:37.338439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:37.338419    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:37.345808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:37.345787    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-wtxmd"]
Jul 17 21:48:38.177759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.177704    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:38.178036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178025    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.178085 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178038    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.178085 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178047    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.178085 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178052    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.178180 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178108    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.178180 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.178116    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:48:38.182479 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.182461    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:38.188274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.188247    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:38.233128 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.233097    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt\" (UID: \"8ab9073a-6c6d-443b-819d-68e0e965eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:38.334131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.334091    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt\" (UID: \"8ab9073a-6c6d-443b-819d-68e0e965eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:38.334822 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.334800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt\" (UID: \"8ab9073a-6c6d-443b-819d-68e0e965eeb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:38.494190 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.494158    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:38.536635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.536590    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="795f1089-f704-4827-a178-a9d53330c397" path="/var/lib/kubelet/pods/795f1089-f704-4827-a178-a9d53330c397/volumes"
Jul 17 21:48:38.635118 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:38.635096    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:38.637740 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:48:38.637680    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ab9073a_6c6d_443b_819d_68e0e965eeb3.slice/crio-b9e4d74351ab381b3748f2c05ca0c1fe50c11c148a37615966ff636cdb25be25 WatchSource:0}: Error finding container b9e4d74351ab381b3748f2c05ca0c1fe50c11c148a37615966ff636cdb25be25: Status 404 returned error can't find the container with id b9e4d74351ab381b3748f2c05ca0c1fe50c11c148a37615966ff636cdb25be25
Jul 17 21:48:39.332848 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:39.332812    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerStarted","Data":"76922de10dff83e16479d48d6b558ff52fe081d22b8a7e17f81510cc4fe416b3"}
Jul 17 21:48:39.333031 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:39.332865    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerStarted","Data":"b9e4d74351ab381b3748f2c05ca0c1fe50c11c148a37615966ff636cdb25be25"}
Jul 17 21:48:39.348949 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:39.348901    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" podStartSLOduration=1.348885763 podStartE2EDuration="1.348885763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:48:39.347858664 +0000 UTC m=+1895.328229143" watchObservedRunningTime="2026-07-17 21:48:39.348885763 +0000 UTC m=+1895.329256240"
Jul 17 21:48:40.339385 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:40.339347    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerID="76922de10dff83e16479d48d6b558ff52fe081d22b8a7e17f81510cc4fe416b3" exitCode=6
Jul 17 21:48:40.339817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:40.339400    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerDied","Data":"76922de10dff83e16479d48d6b558ff52fe081d22b8a7e17f81510cc4fe416b3"}
Jul 17 21:48:40.339817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:40.339610    2546 scope.go:117] "RemoveContainer" containerID="76922de10dff83e16479d48d6b558ff52fe081d22b8a7e17f81510cc4fe416b3"
Jul 17 21:48:41.345536 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:41.345504    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerID="f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9" exitCode=6
Jul 17 21:48:41.346012 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:41.345585    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerDied","Data":"f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9"}
Jul 17 21:48:41.346012 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:41.345644    2546 scope.go:117] "RemoveContainer" containerID="76922de10dff83e16479d48d6b558ff52fe081d22b8a7e17f81510cc4fe416b3"
Jul 17 21:48:41.346012 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:41.345867    2546 scope.go:117] "RemoveContainer" containerID="f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9"
Jul 17 21:48:41.346144 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:41.346110    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt_odh-ai-gateway-infra(8ab9073a-6c6d-443b-819d-68e0e965eeb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3"
Jul 17 21:48:42.352386 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:42.352362    2546 scope.go:117] "RemoveContainer" containerID="f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9"
Jul 17 21:48:42.352784 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:42.352528    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt_odh-ai-gateway-infra(8ab9073a-6c6d-443b-819d-68e0e965eeb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3"
Jul 17 21:48:44.538217 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:44.535997    2546 scope.go:117] "RemoveContainer" containerID="6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1"
Jul 17 21:48:45.368254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:45.368220    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerID="2e0904354c6b2e78115a6f700c8969a377f3407cd2da7cc71085f4b1b6c1bab0" exitCode=6
Jul 17 21:48:45.368428 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:45.368292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" event={"ID":"e5c417a9-856a-4c3a-b90b-a911eee90ecf","Type":"ContainerDied","Data":"2e0904354c6b2e78115a6f700c8969a377f3407cd2da7cc71085f4b1b6c1bab0"}
Jul 17 21:48:45.368428 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:45.368353    2546 scope.go:117] "RemoveContainer" containerID="6b593e91e93ce1a028820b13db7a70b8ee53b222b9f3cb6989d263a8552312d1"
Jul 17 21:48:45.368716 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:45.368698    2546 scope.go:117] "RemoveContainer" containerID="2e0904354c6b2e78115a6f700c8969a377f3407cd2da7cc71085f4b1b6c1bab0"
Jul 17 21:48:45.368936 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:45.368916    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s_odh-ai-gateway-infra(e5c417a9-856a-4c3a-b90b-a911eee90ecf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf"
Jul 17 21:48:46.397132 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:46.397098    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:46.530493 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:46.530467    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:46.602681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:46.602648    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca\") pod \"e5c417a9-856a-4c3a-b90b-a911eee90ecf\" (UID: \"e5c417a9-856a-4c3a-b90b-a911eee90ecf\") "
Jul 17 21:48:46.603013 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:46.602990    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5c417a9-856a-4c3a-b90b-a911eee90ecf" (UID: "e5c417a9-856a-4c3a-b90b-a911eee90ecf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:46.704157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:46.704068    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c417a9-856a-4c3a-b90b-a911eee90ecf-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:47.380793 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:47.380756    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s" event={"ID":"e5c417a9-856a-4c3a-b90b-a911eee90ecf","Type":"ContainerDied","Data":"2dade5a8eb22970b1fa16e07977c95d63f781fee070f0ba9348ff85c2defdb95"}
Jul 17 21:48:47.381059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:47.380827    2546 scope.go:117] "RemoveContainer" containerID="2e0904354c6b2e78115a6f700c8969a377f3407cd2da7cc71085f4b1b6c1bab0"
Jul 17 21:48:47.381059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:47.380768    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"
Jul 17 21:48:47.401775 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:47.401746    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:47.410827 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:47.410804    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-xc44s"]
Jul 17 21:48:48.536637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:48.536601    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" path="/var/lib/kubelet/pods/e5c417a9-856a-4c3a-b90b-a911eee90ecf/volumes"
Jul 17 21:48:54.533334 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:54.533303    2546 scope.go:117] "RemoveContainer" containerID="f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9"
Jul 17 21:48:55.425206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:55.425123    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerID="432634d3b310b62c6c1abf598a80d80fa4caf473a6c0850e6c94dcc73dd636cb" exitCode=6
Jul 17 21:48:55.425333 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:55.425211    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerDied","Data":"432634d3b310b62c6c1abf598a80d80fa4caf473a6c0850e6c94dcc73dd636cb"}
Jul 17 21:48:55.425333 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:55.425282    2546 scope.go:117] "RemoveContainer" containerID="f9b3338d81058a7ae7b3fcb2ac2525d336adacf0a4dbe719fc0820b3f52b7ef9"
Jul 17 21:48:55.425593 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:55.425564    2546 scope.go:117] "RemoveContainer" containerID="432634d3b310b62c6c1abf598a80d80fa4caf473a6c0850e6c94dcc73dd636cb"
Jul 17 21:48:55.425814 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:48:55.425790    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt_odh-ai-gateway-infra(8ab9073a-6c6d-443b-819d-68e0e965eeb3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3"
Jul 17 21:48:56.451380 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:56.451347    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:56.581371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:56.581343    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:56.685108 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:56.685077    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca\") pod \"8ab9073a-6c6d-443b-819d-68e0e965eeb3\" (UID: \"8ab9073a-6c6d-443b-819d-68e0e965eeb3\") "
Jul 17 21:48:56.685565 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:56.685441    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ab9073a-6c6d-443b-819d-68e0e965eeb3" (UID: "8ab9073a-6c6d-443b-819d-68e0e965eeb3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:48:56.786271 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:56.786232    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ab9073a-6c6d-443b-819d-68e0e965eeb3-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:48:57.437649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:57.437613    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt" event={"ID":"8ab9073a-6c6d-443b-819d-68e0e965eeb3","Type":"ContainerDied","Data":"b9e4d74351ab381b3748f2c05ca0c1fe50c11c148a37615966ff636cdb25be25"}
Jul 17 21:48:57.437649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:57.437641    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"
Jul 17 21:48:57.437919 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:57.437686    2546 scope.go:117] "RemoveContainer" containerID="432634d3b310b62c6c1abf598a80d80fa4caf473a6c0850e6c94dcc73dd636cb"
Jul 17 21:48:57.459571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:57.459545    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:57.465209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:57.465186    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-m65rt"]
Jul 17 21:48:58.535916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:48:58.535889    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" path="/var/lib/kubelet/pods/8ab9073a-6c6d-443b-819d-68e0e965eeb3/volumes"
Jul 17 21:49:17.228159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228126    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228421    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228431    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228440    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228445    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228455    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228460    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228475    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228481    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228489    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228494    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228501    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228506    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228553    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228561    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="795f1089-f704-4827-a178-a9d53330c397" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228568    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228573    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.228607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.228578    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:17.231506 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.231487    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:17.233877 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.233857    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:49:17.238427 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.238401    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:17.346465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.346434    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd\" (UID: \"eef01b00-6c7a-4ddb-89f3-39d036514f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:17.447548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.447526    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd\" (UID: \"eef01b00-6c7a-4ddb-89f3-39d036514f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:17.448143 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.448126    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd\" (UID: \"eef01b00-6c7a-4ddb-89f3-39d036514f9b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:17.543554 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.543536    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:17.673782 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:17.673758    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:17.675847 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:49:17.675821    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeef01b00_6c7a_4ddb_89f3_39d036514f9b.slice/crio-97645bfe90157c07f7a38345d3db107b88f457783069483617c923ae8e70dc2c WatchSource:0}: Error finding container 97645bfe90157c07f7a38345d3db107b88f457783069483617c923ae8e70dc2c: Status 404 returned error can't find the container with id 97645bfe90157c07f7a38345d3db107b88f457783069483617c923ae8e70dc2c
Jul 17 21:49:18.546179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:18.546090    2546 generic.go:356] "Generic (PLEG): container finished" podID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerID="0688b63ebafc6b7da404e124a9e347ceccba4e7887470bda601313a6cf4cf723" exitCode=7
Jul 17 21:49:18.546179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:18.546144    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" event={"ID":"eef01b00-6c7a-4ddb-89f3-39d036514f9b","Type":"ContainerDied","Data":"0688b63ebafc6b7da404e124a9e347ceccba4e7887470bda601313a6cf4cf723"}
Jul 17 21:49:18.546574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:18.546197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" event={"ID":"eef01b00-6c7a-4ddb-89f3-39d036514f9b","Type":"ContainerStarted","Data":"97645bfe90157c07f7a38345d3db107b88f457783069483617c923ae8e70dc2c"}
Jul 17 21:49:18.546574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:18.546362    2546 scope.go:117] "RemoveContainer" containerID="0688b63ebafc6b7da404e124a9e347ceccba4e7887470bda601313a6cf4cf723"
Jul 17 21:49:19.552888 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:19.552856    2546 generic.go:356] "Generic (PLEG): container finished" podID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerID="d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237" exitCode=7
Jul 17 21:49:19.553274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:19.552942    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" event={"ID":"eef01b00-6c7a-4ddb-89f3-39d036514f9b","Type":"ContainerDied","Data":"d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237"}
Jul 17 21:49:19.553274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:19.552995    2546 scope.go:117] "RemoveContainer" containerID="0688b63ebafc6b7da404e124a9e347ceccba4e7887470bda601313a6cf4cf723"
Jul 17 21:49:19.553274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:19.553096    2546 scope.go:117] "RemoveContainer" containerID="d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237"
Jul 17 21:49:19.553412 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:19.553312    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd_odh-ai-gateway-infra(eef01b00-6c7a-4ddb-89f3-39d036514f9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b"
Jul 17 21:49:20.051182 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.051149    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:20.051492 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.051477    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:20.051538 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.051494    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:20.051574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.051556    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ab9073a-6c6d-443b-819d-68e0e965eeb3" containerName="revoke-keys"
Jul 17 21:49:20.051574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.051564    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c417a9-856a-4c3a-b90b-a911eee90ecf" containerName="revoke-keys"
Jul 17 21:49:20.055620 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.055587    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:20.064149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.064123    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:20.168705 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.168679    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg\" (UID: \"1b96517e-fd56-4b52-abc9-e6d11793366a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:20.270044 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.270015    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg\" (UID: \"1b96517e-fd56-4b52-abc9-e6d11793366a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:20.270649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.270629    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg\" (UID: \"1b96517e-fd56-4b52-abc9-e6d11793366a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:20.367825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.367766    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:20.510565 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.510540    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:20.512966 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:49:20.512935    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b96517e_fd56_4b52_abc9_e6d11793366a.slice/crio-b917a9c462667311b0ff5c7454e5e0f5ac2cbb1304a45c76d66a7c5e63d3717d WatchSource:0}: Error finding container b917a9c462667311b0ff5c7454e5e0f5ac2cbb1304a45c76d66a7c5e63d3717d: Status 404 returned error can't find the container with id b917a9c462667311b0ff5c7454e5e0f5ac2cbb1304a45c76d66a7c5e63d3717d
Jul 17 21:49:20.559004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.558987    2546 scope.go:117] "RemoveContainer" containerID="d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237"
Jul 17 21:49:20.559356 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:20.559234    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd_odh-ai-gateway-infra(eef01b00-6c7a-4ddb-89f3-39d036514f9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b"
Jul 17 21:49:20.560075 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:20.560052    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" event={"ID":"1b96517e-fd56-4b52-abc9-e6d11793366a","Type":"ContainerStarted","Data":"b917a9c462667311b0ff5c7454e5e0f5ac2cbb1304a45c76d66a7c5e63d3717d"}
Jul 17 21:49:21.566905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:21.566873    2546 generic.go:356] "Generic (PLEG): container finished" podID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerID="a7b881335218bb4843d3639ab3f7d45a4af2797bc357f662df006e6edeb96e8d" exitCode=6
Jul 17 21:49:21.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:21.566958    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" event={"ID":"1b96517e-fd56-4b52-abc9-e6d11793366a","Type":"ContainerDied","Data":"a7b881335218bb4843d3639ab3f7d45a4af2797bc357f662df006e6edeb96e8d"}
Jul 17 21:49:21.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:21.567104    2546 scope.go:117] "RemoveContainer" containerID="a7b881335218bb4843d3639ab3f7d45a4af2797bc357f662df006e6edeb96e8d"
Jul 17 21:49:22.572899 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:22.572831    2546 generic.go:356] "Generic (PLEG): container finished" podID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerID="105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb" exitCode=6
Jul 17 21:49:22.573279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:22.572930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" event={"ID":"1b96517e-fd56-4b52-abc9-e6d11793366a","Type":"ContainerDied","Data":"105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb"}
Jul 17 21:49:22.573279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:22.572999    2546 scope.go:117] "RemoveContainer" containerID="a7b881335218bb4843d3639ab3f7d45a4af2797bc357f662df006e6edeb96e8d"
Jul 17 21:49:22.573279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:22.573069    2546 scope.go:117] "RemoveContainer" containerID="105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb"
Jul 17 21:49:22.573279 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:22.573237    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg_odh-ai-gateway-infra(1b96517e-fd56-4b52-abc9-e6d11793366a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a"
Jul 17 21:49:23.579867 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:23.579839    2546 scope.go:117] "RemoveContainer" containerID="105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb"
Jul 17 21:49:23.580229 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:23.580002    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg_odh-ai-gateway-infra(1b96517e-fd56-4b52-abc9-e6d11793366a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a"
Jul 17 21:49:28.222796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.222756    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:28.227195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.227179    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:28.233452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.233425    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:28.330013 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.329981    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg\" (UID: \"75e279f9-01f5-4a4e-8275-fa36509b2303\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:28.431160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.431128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg\" (UID: \"75e279f9-01f5-4a4e-8275-fa36509b2303\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:28.431775 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.431753    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg\" (UID: \"75e279f9-01f5-4a4e-8275-fa36509b2303\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:28.540268 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.540244    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:28.667588 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:28.667562    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:28.669630 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:49:28.669605    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75e279f9_01f5_4a4e_8275_fa36509b2303.slice/crio-f8bd2017eba5ec1d63e4a1a9c8448fed7b6aadab35a6bbd6bc8256d044fbc77c WatchSource:0}: Error finding container f8bd2017eba5ec1d63e4a1a9c8448fed7b6aadab35a6bbd6bc8256d044fbc77c: Status 404 returned error can't find the container with id f8bd2017eba5ec1d63e4a1a9c8448fed7b6aadab35a6bbd6bc8256d044fbc77c
Jul 17 21:49:29.611995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:29.611913    2546 generic.go:356] "Generic (PLEG): container finished" podID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerID="ff9d3dad381f6b723a61db303da1268147bdb7863ceaecb967df41079df3217e" exitCode=6
Jul 17 21:49:29.611995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:29.611950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" event={"ID":"75e279f9-01f5-4a4e-8275-fa36509b2303","Type":"ContainerDied","Data":"ff9d3dad381f6b723a61db303da1268147bdb7863ceaecb967df41079df3217e"}
Jul 17 21:49:29.611995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:29.611987    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" event={"ID":"75e279f9-01f5-4a4e-8275-fa36509b2303","Type":"ContainerStarted","Data":"f8bd2017eba5ec1d63e4a1a9c8448fed7b6aadab35a6bbd6bc8256d044fbc77c"}
Jul 17 21:49:29.612416 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:29.612137    2546 scope.go:117] "RemoveContainer" containerID="ff9d3dad381f6b723a61db303da1268147bdb7863ceaecb967df41079df3217e"
Jul 17 21:49:30.618399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:30.618364    2546 generic.go:356] "Generic (PLEG): container finished" podID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerID="840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930" exitCode=6
Jul 17 21:49:30.618828 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:30.618449    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" event={"ID":"75e279f9-01f5-4a4e-8275-fa36509b2303","Type":"ContainerDied","Data":"840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930"}
Jul 17 21:49:30.618828 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:30.618514    2546 scope.go:117] "RemoveContainer" containerID="ff9d3dad381f6b723a61db303da1268147bdb7863ceaecb967df41079df3217e"
Jul 17 21:49:30.618828 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:30.618630    2546 scope.go:117] "RemoveContainer" containerID="840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930"
Jul 17 21:49:30.618969 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:30.618864    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg_odh-ai-gateway-infra(75e279f9-01f5-4a4e-8275-fa36509b2303)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303"
Jul 17 21:49:31.531382 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:31.531354    2546 scope.go:117] "RemoveContainer" containerID="d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237"
Jul 17 21:49:31.626052 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:31.626033    2546 scope.go:117] "RemoveContainer" containerID="840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930"
Jul 17 21:49:31.626329 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:31.626222    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg_odh-ai-gateway-infra(75e279f9-01f5-4a4e-8275-fa36509b2303)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303"
Jul 17 21:49:32.632933 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:32.632900    2546 generic.go:356] "Generic (PLEG): container finished" podID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerID="578856e73dc01deacb613588777c9f1d5b86958ee997b3f3a5cc78f604689b1f" exitCode=7
Jul 17 21:49:32.633360 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:32.632974    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" event={"ID":"eef01b00-6c7a-4ddb-89f3-39d036514f9b","Type":"ContainerDied","Data":"578856e73dc01deacb613588777c9f1d5b86958ee997b3f3a5cc78f604689b1f"}
Jul 17 21:49:32.633360 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:32.633046    2546 scope.go:117] "RemoveContainer" containerID="d7109221a9e19ae328d9df87b54a1c34450cf57148b9609bec568a0301cc9237"
Jul 17 21:49:32.633360 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:32.633327    2546 scope.go:117] "RemoveContainer" containerID="578856e73dc01deacb613588777c9f1d5b86958ee997b3f3a5cc78f604689b1f"
Jul 17 21:49:32.633540 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:32.633516    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd_odh-ai-gateway-infra(eef01b00-6c7a-4ddb-89f3-39d036514f9b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b"
Jul 17 21:49:33.658025 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:33.657994    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:33.786901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:33.786878    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:33.871026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:33.870999    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca\") pod \"eef01b00-6c7a-4ddb-89f3-39d036514f9b\" (UID: \"eef01b00-6c7a-4ddb-89f3-39d036514f9b\") "
Jul 17 21:49:33.871348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:33.871328    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eef01b00-6c7a-4ddb-89f3-39d036514f9b" (UID: "eef01b00-6c7a-4ddb-89f3-39d036514f9b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:33.972530 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:33.972474    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eef01b00-6c7a-4ddb-89f3-39d036514f9b-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:49:34.645644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:34.645611    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd" event={"ID":"eef01b00-6c7a-4ddb-89f3-39d036514f9b","Type":"ContainerDied","Data":"97645bfe90157c07f7a38345d3db107b88f457783069483617c923ae8e70dc2c"}
Jul 17 21:49:34.645644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:34.645635    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"
Jul 17 21:49:34.645867 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:34.645671    2546 scope.go:117] "RemoveContainer" containerID="578856e73dc01deacb613588777c9f1d5b86958ee997b3f3a5cc78f604689b1f"
Jul 17 21:49:34.662489 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:34.662469    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:34.668867 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:34.668844    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k86hd"]
Jul 17 21:49:36.531164 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:36.531122    2546 scope.go:117] "RemoveContainer" containerID="105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb"
Jul 17 21:49:36.537980 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:36.537951    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" path="/var/lib/kubelet/pods/eef01b00-6c7a-4ddb-89f3-39d036514f9b/volumes"
Jul 17 21:49:37.669765 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:37.669668    2546 generic.go:356] "Generic (PLEG): container finished" podID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerID="2962d4278a8ebf1a80df1f742e81c525602f8b6f8ea87979aba3b7cc6ae4c944" exitCode=6
Jul 17 21:49:37.669765 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:37.669748    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" event={"ID":"1b96517e-fd56-4b52-abc9-e6d11793366a","Type":"ContainerDied","Data":"2962d4278a8ebf1a80df1f742e81c525602f8b6f8ea87979aba3b7cc6ae4c944"}
Jul 17 21:49:37.670209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:37.669842    2546 scope.go:117] "RemoveContainer" containerID="105cdda8be6b998be5e16af094b521db1b5d762830da5af7f2147d8ac5faedcb"
Jul 17 21:49:37.670209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:37.670190    2546 scope.go:117] "RemoveContainer" containerID="2962d4278a8ebf1a80df1f742e81c525602f8b6f8ea87979aba3b7cc6ae4c944"
Jul 17 21:49:37.670436 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:37.670413    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg_odh-ai-gateway-infra(1b96517e-fd56-4b52-abc9-e6d11793366a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a"
Jul 17 21:49:38.698741 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:38.698693    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:38.837649 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:38.837627    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:38.909147 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:38.909119    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca\") pod \"1b96517e-fd56-4b52-abc9-e6d11793366a\" (UID: \"1b96517e-fd56-4b52-abc9-e6d11793366a\") "
Jul 17 21:49:38.909448 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:38.909422    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1b96517e-fd56-4b52-abc9-e6d11793366a" (UID: "1b96517e-fd56-4b52-abc9-e6d11793366a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:39.009603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.009577    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b96517e-fd56-4b52-abc9-e6d11793366a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:49:39.682187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.682159    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"
Jul 17 21:49:39.682330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.682206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg" event={"ID":"1b96517e-fd56-4b52-abc9-e6d11793366a","Type":"ContainerDied","Data":"b917a9c462667311b0ff5c7454e5e0f5ac2cbb1304a45c76d66a7c5e63d3717d"}
Jul 17 21:49:39.682330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.682262    2546 scope.go:117] "RemoveContainer" containerID="2962d4278a8ebf1a80df1f742e81c525602f8b6f8ea87979aba3b7cc6ae4c944"
Jul 17 21:49:39.703611 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.703578    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:39.708893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:39.708870    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-6wncg"]
Jul 17 21:49:40.536768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:40.536711    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" path="/var/lib/kubelet/pods/1b96517e-fd56-4b52-abc9-e6d11793366a/volumes"
Jul 17 21:49:43.530837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:43.530807    2546 scope.go:117] "RemoveContainer" containerID="840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930"
Jul 17 21:49:44.709433 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:44.709399    2546 generic.go:356] "Generic (PLEG): container finished" podID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerID="965fc8722c885578248532b178f6c174c92a9285091a87c6bb527f890afd87c7" exitCode=6
Jul 17 21:49:44.709905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:44.709479    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" event={"ID":"75e279f9-01f5-4a4e-8275-fa36509b2303","Type":"ContainerDied","Data":"965fc8722c885578248532b178f6c174c92a9285091a87c6bb527f890afd87c7"}
Jul 17 21:49:44.709905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:44.709542    2546 scope.go:117] "RemoveContainer" containerID="840187edf7628637f937f0e517cb6d7c41aed0cf6c3af76af078595ec32d4930"
Jul 17 21:49:44.709992 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:44.709958    2546 scope.go:117] "RemoveContainer" containerID="965fc8722c885578248532b178f6c174c92a9285091a87c6bb527f890afd87c7"
Jul 17 21:49:44.710157 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:49:44.710136    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg_odh-ai-gateway-infra(75e279f9-01f5-4a4e-8275-fa36509b2303)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303"
Jul 17 21:49:45.735709 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:45.735672    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:45.864928 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:45.864903    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:45.961852 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:45.961827    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca\") pod \"75e279f9-01f5-4a4e-8275-fa36509b2303\" (UID: \"75e279f9-01f5-4a4e-8275-fa36509b2303\") "
Jul 17 21:49:45.962147 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:45.962127    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75e279f9-01f5-4a4e-8275-fa36509b2303" (UID: "75e279f9-01f5-4a4e-8275-fa36509b2303"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:49:46.062340 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.062264    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75e279f9-01f5-4a4e-8275-fa36509b2303-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:49:46.722066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.722028    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg" event={"ID":"75e279f9-01f5-4a4e-8275-fa36509b2303","Type":"ContainerDied","Data":"f8bd2017eba5ec1d63e4a1a9c8448fed7b6aadab35a6bbd6bc8256d044fbc77c"}
Jul 17 21:49:46.722254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.722091    2546 scope.go:117] "RemoveContainer" containerID="965fc8722c885578248532b178f6c174c92a9285091a87c6bb527f890afd87c7"
Jul 17 21:49:46.722254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.722050    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"
Jul 17 21:49:46.742815 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.740422    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:46.748403 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:46.748374    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-wfqdg"]
Jul 17 21:49:48.536700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:49:48.536670    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" path="/var/lib/kubelet/pods/75e279f9-01f5-4a4e-8275-fa36509b2303/volumes"
Jul 17 21:50:07.272612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.272573    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.272998    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273015    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273029    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273038    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273051    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273059    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273068    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273075    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273087    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273096    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273103    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273111    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273132    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273141    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273228    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273238    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273247    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273257    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273266    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273275    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273283    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b96517e-fd56-4b52-abc9-e6d11793366a" containerName="revoke-keys"
Jul 17 21:50:07.273714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.273294    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:07.276452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.276432    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:07.278932 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.278910    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:50:07.285383 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.285346    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:07.338110 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.338086    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l\" (UID: \"cabe8471-1e02-48eb-a33d-28900edaccd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:07.438481 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.438454    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l\" (UID: \"cabe8471-1e02-48eb-a33d-28900edaccd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:07.439114 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.439094    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l\" (UID: \"cabe8471-1e02-48eb-a33d-28900edaccd9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:07.587663 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.587607    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:07.720451 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.720429    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:07.722599 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:50:07.722568    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabe8471_1e02_48eb_a33d_28900edaccd9.slice/crio-b146f818bb814b8eef650b35ec15908583975f6ae97622847e4444f87c7d67e3 WatchSource:0}: Error finding container b146f818bb814b8eef650b35ec15908583975f6ae97622847e4444f87c7d67e3: Status 404 returned error can't find the container with id b146f818bb814b8eef650b35ec15908583975f6ae97622847e4444f87c7d67e3
Jul 17 21:50:07.828414 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:07.828383    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerStarted","Data":"b146f818bb814b8eef650b35ec15908583975f6ae97622847e4444f87c7d67e3"}
Jul 17 21:50:08.836320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:08.836282    2546 generic.go:356] "Generic (PLEG): container finished" podID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerID="4ab7dbde312c916be2c715fc535ca8d5f2fd3ae436bf5cd3aed8b20175f92f02" exitCode=7
Jul 17 21:50:08.836715 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:08.836389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerDied","Data":"4ab7dbde312c916be2c715fc535ca8d5f2fd3ae436bf5cd3aed8b20175f92f02"}
Jul 17 21:50:08.836715 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:08.836558    2546 scope.go:117] "RemoveContainer" containerID="4ab7dbde312c916be2c715fc535ca8d5f2fd3ae436bf5cd3aed8b20175f92f02"
Jul 17 21:50:09.843136 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:09.843103    2546 generic.go:356] "Generic (PLEG): container finished" podID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerID="48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9" exitCode=7
Jul 17 21:50:09.843590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:09.843197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerDied","Data":"48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9"}
Jul 17 21:50:09.843590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:09.843312    2546 scope.go:117] "RemoveContainer" containerID="4ab7dbde312c916be2c715fc535ca8d5f2fd3ae436bf5cd3aed8b20175f92f02"
Jul 17 21:50:09.843590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:09.843358    2546 scope.go:117] "RemoveContainer" containerID="48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9"
Jul 17 21:50:09.843590 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:09.843548    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l_odh-ai-gateway-infra(cabe8471-1e02-48eb-a33d-28900edaccd9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9"
Jul 17 21:50:10.099366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099289    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:10.099634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099618    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:10.099682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099635    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eef01b00-6c7a-4ddb-89f3-39d036514f9b" containerName="revoke-keys"
Jul 17 21:50:10.099682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099651    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:10.099682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099657    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:10.099830 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.099775    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="75e279f9-01f5-4a4e-8275-fa36509b2303" containerName="revoke-keys"
Jul 17 21:50:10.104005 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.103986    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:10.110049 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.110024    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:10.260386 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.260348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88\" (UID: \"8364b43f-09f0-4a0b-811f-c01ad2e5356b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:10.361437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.361356    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88\" (UID: \"8364b43f-09f0-4a0b-811f-c01ad2e5356b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:10.361997 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.361977    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88\" (UID: \"8364b43f-09f0-4a0b-811f-c01ad2e5356b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:10.416256 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.416231    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:10.544253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.544233    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:10.546253 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:50:10.546225    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8364b43f_09f0_4a0b_811f_c01ad2e5356b.slice/crio-f8dcd1dccd3224b1d7b63bc4f35edc0f3f10fbafcce98dd6a98e00ddfc834411 WatchSource:0}: Error finding container f8dcd1dccd3224b1d7b63bc4f35edc0f3f10fbafcce98dd6a98e00ddfc834411: Status 404 returned error can't find the container with id f8dcd1dccd3224b1d7b63bc4f35edc0f3f10fbafcce98dd6a98e00ddfc834411
Jul 17 21:50:10.851905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.851872    2546 scope.go:117] "RemoveContainer" containerID="48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9"
Jul 17 21:50:10.852301 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:10.852093    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l_odh-ai-gateway-infra(cabe8471-1e02-48eb-a33d-28900edaccd9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9"
Jul 17 21:50:10.853186 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.853162    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerStarted","Data":"27423c4685709182fa4ca32ac07355c86fb83aa62ceca4c46ac4ad509a95dfae"}
Jul 17 21:50:10.853293 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.853216    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerStarted","Data":"f8dcd1dccd3224b1d7b63bc4f35edc0f3f10fbafcce98dd6a98e00ddfc834411"}
Jul 17 21:50:10.886158 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:10.886112    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" podStartSLOduration=0.886099291 podStartE2EDuration="886.099291ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:10.8836726 +0000 UTC m=+1986.864043078" watchObservedRunningTime="2026-07-17 21:50:10.886099291 +0000 UTC m=+1986.866469767"
Jul 17 21:50:11.860151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:11.860117    2546 generic.go:356] "Generic (PLEG): container finished" podID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerID="27423c4685709182fa4ca32ac07355c86fb83aa62ceca4c46ac4ad509a95dfae" exitCode=6
Jul 17 21:50:11.860151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:11.860154    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerDied","Data":"27423c4685709182fa4ca32ac07355c86fb83aa62ceca4c46ac4ad509a95dfae"}
Jul 17 21:50:11.860606 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:11.860395    2546 scope.go:117] "RemoveContainer" containerID="27423c4685709182fa4ca32ac07355c86fb83aa62ceca4c46ac4ad509a95dfae"
Jul 17 21:50:12.867278 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:12.867240    2546 generic.go:356] "Generic (PLEG): container finished" podID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerID="a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739" exitCode=6
Jul 17 21:50:12.867824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:12.867327    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerDied","Data":"a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739"}
Jul 17 21:50:12.867824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:12.867390    2546 scope.go:117] "RemoveContainer" containerID="27423c4685709182fa4ca32ac07355c86fb83aa62ceca4c46ac4ad509a95dfae"
Jul 17 21:50:12.867824 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:12.867631    2546 scope.go:117] "RemoveContainer" containerID="a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739"
Jul 17 21:50:12.867993 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:12.867881    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88_odh-ai-gateway-infra(8364b43f-09f0-4a0b-811f-c01ad2e5356b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b"
Jul 17 21:50:13.874623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:13.874596    2546 scope.go:117] "RemoveContainer" containerID="a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739"
Jul 17 21:50:13.875060 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:13.874807    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88_odh-ai-gateway-infra(8364b43f-09f0-4a0b-811f-c01ad2e5356b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b"
Jul 17 21:50:18.272887 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.272851    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:18.277534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.277515    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:18.283063 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.283018    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:18.426019 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.425987    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd\" (UID: \"92a90f08-53c7-4111-8b76-ea38aa833181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:18.526919 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.526847    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd\" (UID: \"92a90f08-53c7-4111-8b76-ea38aa833181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:18.527436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.527415    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd\" (UID: \"92a90f08-53c7-4111-8b76-ea38aa833181\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:18.590146 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.590122    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:18.721488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.721461    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:18.723635 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:50:18.723605    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a90f08_53c7_4111_8b76_ea38aa833181.slice/crio-aab6dc41de5a223010a6508b8457d834ed80d78d316fa21b5aa156311887e104 WatchSource:0}: Error finding container aab6dc41de5a223010a6508b8457d834ed80d78d316fa21b5aa156311887e104: Status 404 returned error can't find the container with id aab6dc41de5a223010a6508b8457d834ed80d78d316fa21b5aa156311887e104
Jul 17 21:50:18.899770 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:18.899672    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerStarted","Data":"aab6dc41de5a223010a6508b8457d834ed80d78d316fa21b5aa156311887e104"}
Jul 17 21:50:19.905970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:19.905929    2546 generic.go:356] "Generic (PLEG): container finished" podID="92a90f08-53c7-4111-8b76-ea38aa833181" containerID="8b354a65f260da7af55b617e57400a20c4010c90036baddff6252fe285f1140b" exitCode=6
Jul 17 21:50:19.906528 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:19.906010    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerDied","Data":"8b354a65f260da7af55b617e57400a20c4010c90036baddff6252fe285f1140b"}
Jul 17 21:50:19.906528 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:19.906222    2546 scope.go:117] "RemoveContainer" containerID="8b354a65f260da7af55b617e57400a20c4010c90036baddff6252fe285f1140b"
Jul 17 21:50:20.912892 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:20.912807    2546 generic.go:356] "Generic (PLEG): container finished" podID="92a90f08-53c7-4111-8b76-ea38aa833181" containerID="fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56" exitCode=6
Jul 17 21:50:20.913283 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:20.912886    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerDied","Data":"fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56"}
Jul 17 21:50:20.913283 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:20.912944    2546 scope.go:117] "RemoveContainer" containerID="8b354a65f260da7af55b617e57400a20c4010c90036baddff6252fe285f1140b"
Jul 17 21:50:20.913283 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:20.913174    2546 scope.go:117] "RemoveContainer" containerID="fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56"
Jul 17 21:50:20.913413 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:20.913379    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd_odh-ai-gateway-infra(92a90f08-53c7-4111-8b76-ea38aa833181)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" podUID="92a90f08-53c7-4111-8b76-ea38aa833181"
Jul 17 21:50:21.920088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:21.920054    2546 scope.go:117] "RemoveContainer" containerID="fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56"
Jul 17 21:50:21.920504 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:21.920277    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd_odh-ai-gateway-infra(92a90f08-53c7-4111-8b76-ea38aa833181)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" podUID="92a90f08-53c7-4111-8b76-ea38aa833181"
Jul 17 21:50:24.534734 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:24.534685    2546 scope.go:117] "RemoveContainer" containerID="a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739"
Jul 17 21:50:24.535238 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:24.535116    2546 scope.go:117] "RemoveContainer" containerID="48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9"
Jul 17 21:50:24.936201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:24.936176    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerStarted","Data":"81b58d9bfb406a50863b79c874ce05b4bf05bb50d9a37f0cacf4ae248ae2a50b"}
Jul 17 21:50:24.937846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:24.937826    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerStarted","Data":"fe1d99d3aa84a6c709e49fd50fcef787fb57a908fceb699105783d855b4bcf6b"}
Jul 17 21:50:24.954022 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:24.953977    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" podStartSLOduration=17.953963213 podStartE2EDuration="17.953963213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:24.952452247 +0000 UTC m=+2000.932822725" watchObservedRunningTime="2026-07-17 21:50:24.953963213 +0000 UTC m=+2000.934333743"
Jul 17 21:50:25.944190 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.944157    2546 generic.go:356] "Generic (PLEG): container finished" podID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerID="81b58d9bfb406a50863b79c874ce05b4bf05bb50d9a37f0cacf4ae248ae2a50b" exitCode=7
Jul 17 21:50:25.944692 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.944235    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerDied","Data":"81b58d9bfb406a50863b79c874ce05b4bf05bb50d9a37f0cacf4ae248ae2a50b"}
Jul 17 21:50:25.944692 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.944302    2546 scope.go:117] "RemoveContainer" containerID="48849ad5c74c4209fbb9bee823e5c496029c9134e02617259fc9e1196908c5a9"
Jul 17 21:50:25.944692 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.944614    2546 scope.go:117] "RemoveContainer" containerID="81b58d9bfb406a50863b79c874ce05b4bf05bb50d9a37f0cacf4ae248ae2a50b"
Jul 17 21:50:25.945249 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:25.945218    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l_odh-ai-gateway-infra(cabe8471-1e02-48eb-a33d-28900edaccd9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9"
Jul 17 21:50:25.952650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.952626    2546 generic.go:356] "Generic (PLEG): container finished" podID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerID="fe1d99d3aa84a6c709e49fd50fcef787fb57a908fceb699105783d855b4bcf6b" exitCode=6
Jul 17 21:50:25.952798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.952696    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerDied","Data":"fe1d99d3aa84a6c709e49fd50fcef787fb57a908fceb699105783d855b4bcf6b"}
Jul 17 21:50:25.953034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.953018    2546 scope.go:117] "RemoveContainer" containerID="fe1d99d3aa84a6c709e49fd50fcef787fb57a908fceb699105783d855b4bcf6b"
Jul 17 21:50:25.953214 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:25.953195    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88_odh-ai-gateway-infra(8364b43f-09f0-4a0b-811f-c01ad2e5356b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b"
Jul 17 21:50:25.961410 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.961382    2546 scope.go:117] "RemoveContainer" containerID="a467225f571be8de28dd11dc4910829451c702a58ac37818ba83fd8bb42e2739"
Jul 17 21:50:25.963219 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.963199    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:25.980047 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:25.980019    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:27.098951 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.098929    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:27.127575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.127548    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:27.200151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.200131    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca\") pod \"cabe8471-1e02-48eb-a33d-28900edaccd9\" (UID: \"cabe8471-1e02-48eb-a33d-28900edaccd9\") "
Jul 17 21:50:27.200470 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.200447    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cabe8471-1e02-48eb-a33d-28900edaccd9" (UID: "cabe8471-1e02-48eb-a33d-28900edaccd9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:27.301611 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.301580    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca\") pod \"8364b43f-09f0-4a0b-811f-c01ad2e5356b\" (UID: \"8364b43f-09f0-4a0b-811f-c01ad2e5356b\") "
Jul 17 21:50:27.301829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.301812    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cabe8471-1e02-48eb-a33d-28900edaccd9-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:50:27.301918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.301896    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8364b43f-09f0-4a0b-811f-c01ad2e5356b" (UID: "8364b43f-09f0-4a0b-811f-c01ad2e5356b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:27.402398 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.402374    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8364b43f-09f0-4a0b-811f-c01ad2e5356b-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:50:27.968378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.968327    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l" event={"ID":"cabe8471-1e02-48eb-a33d-28900edaccd9","Type":"ContainerDied","Data":"b146f818bb814b8eef650b35ec15908583975f6ae97622847e4444f87c7d67e3"}
Jul 17 21:50:27.968378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.968335    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"
Jul 17 21:50:27.968378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.968385    2546 scope.go:117] "RemoveContainer" containerID="81b58d9bfb406a50863b79c874ce05b4bf05bb50d9a37f0cacf4ae248ae2a50b"
Jul 17 21:50:27.969643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.969624    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"
Jul 17 21:50:27.969708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.969665    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88" event={"ID":"8364b43f-09f0-4a0b-811f-c01ad2e5356b","Type":"ContainerDied","Data":"f8dcd1dccd3224b1d7b63bc4f35edc0f3f10fbafcce98dd6a98e00ddfc834411"}
Jul 17 21:50:27.978928 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.978913    2546 scope.go:117] "RemoveContainer" containerID="fe1d99d3aa84a6c709e49fd50fcef787fb57a908fceb699105783d855b4bcf6b"
Jul 17 21:50:27.996632 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:27.996610    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:28.004475 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:28.004455    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dbk7l"]
Jul 17 21:50:28.017226 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:28.017198    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:28.021289 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:28.021268    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s4b88"]
Jul 17 21:50:28.536069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:28.536031    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" path="/var/lib/kubelet/pods/8364b43f-09f0-4a0b-811f-c01ad2e5356b/volumes"
Jul 17 21:50:28.536544 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:28.536497    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" path="/var/lib/kubelet/pods/cabe8471-1e02-48eb-a33d-28900edaccd9/volumes"
Jul 17 21:50:35.530875 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:35.530834    2546 scope.go:117] "RemoveContainer" containerID="fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56"
Jul 17 21:50:36.014413 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:36.014378    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerStarted","Data":"9287b7dbcfd68b4144ef23d5611658aad8f7490c4322c3aef0184f05b6bc573f"}
Jul 17 21:50:36.031601 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:36.031542    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" podStartSLOduration=18.031523982 podStartE2EDuration="18.031523982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:50:36.029188415 +0000 UTC m=+2012.009558893" watchObservedRunningTime="2026-07-17 21:50:36.031523982 +0000 UTC m=+2012.011894460"
Jul 17 21:50:37.021265 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:37.021231    2546 generic.go:356] "Generic (PLEG): container finished" podID="92a90f08-53c7-4111-8b76-ea38aa833181" containerID="9287b7dbcfd68b4144ef23d5611658aad8f7490c4322c3aef0184f05b6bc573f" exitCode=6
Jul 17 21:50:37.021686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:37.021306    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerDied","Data":"9287b7dbcfd68b4144ef23d5611658aad8f7490c4322c3aef0184f05b6bc573f"}
Jul 17 21:50:37.021686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:37.021369    2546 scope.go:117] "RemoveContainer" containerID="fc407d3c7091f7303273c2ebc57eaa9efe7767d757f752178155a4608426cd56"
Jul 17 21:50:37.021686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:37.021686    2546 scope.go:117] "RemoveContainer" containerID="9287b7dbcfd68b4144ef23d5611658aad8f7490c4322c3aef0184f05b6bc573f"
Jul 17 21:50:37.021952 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:50:37.021931    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd_odh-ai-gateway-infra(92a90f08-53c7-4111-8b76-ea38aa833181)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" podUID="92a90f08-53c7-4111-8b76-ea38aa833181"
Jul 17 21:50:37.045396 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:37.045364    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:38.165142 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:38.165115    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:38.200041 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:38.200006    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca\") pod \"92a90f08-53c7-4111-8b76-ea38aa833181\" (UID: \"92a90f08-53c7-4111-8b76-ea38aa833181\") "
Jul 17 21:50:38.200374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:38.200350    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92a90f08-53c7-4111-8b76-ea38aa833181" (UID: "92a90f08-53c7-4111-8b76-ea38aa833181"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:50:38.301091 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:38.300998    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92a90f08-53c7-4111-8b76-ea38aa833181-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:50:39.035542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:39.035513    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"
Jul 17 21:50:39.035542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:39.035537    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd" event={"ID":"92a90f08-53c7-4111-8b76-ea38aa833181","Type":"ContainerDied","Data":"aab6dc41de5a223010a6508b8457d834ed80d78d316fa21b5aa156311887e104"}
Jul 17 21:50:39.035823 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:39.035603    2546 scope.go:117] "RemoveContainer" containerID="9287b7dbcfd68b4144ef23d5611658aad8f7490c4322c3aef0184f05b6bc573f"
Jul 17 21:50:39.053302 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:39.053271    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:39.059735 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:39.059688    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-778hd"]
Jul 17 21:50:40.536329 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:50:40.536289    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" path="/var/lib/kubelet/pods/92a90f08-53c7-4111-8b76-ea38aa833181/volumes"
Jul 17 21:51:00.146377 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146345    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146668    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146679    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146689    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146694    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146702    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146708    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146737    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146742    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146751    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146756    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146761    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146767    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146773    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146778    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146786    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146791    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146852    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.146854 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146861    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.147675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146867    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.147675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146873    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:00.147675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146878    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:00.147675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146884    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.147675 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.146890    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8364b43f-09f0-4a0b-811f-c01ad2e5356b" containerName="revoke-keys"
Jul 17 21:51:00.151158 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.151142    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:00.153701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.153682    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:51:00.158169 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.158143    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:00.278446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.278410    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2\" (UID: \"25231378-e7ff-43b9-95eb-103b153fbbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:00.378963 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.378934    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2\" (UID: \"25231378-e7ff-43b9-95eb-103b153fbbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:00.379482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.379463    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2\" (UID: \"25231378-e7ff-43b9-95eb-103b153fbbf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:00.463773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.463692    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:00.593079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:00.593055    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:00.595088 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:51:00.595060    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25231378_e7ff_43b9_95eb_103b153fbbf6.slice/crio-12342d19e58c842bf86c84eebaaa4fd19203d6adaaa4b6eaa09e1fd893ce298b WatchSource:0}: Error finding container 12342d19e58c842bf86c84eebaaa4fd19203d6adaaa4b6eaa09e1fd893ce298b: Status 404 returned error can't find the container with id 12342d19e58c842bf86c84eebaaa4fd19203d6adaaa4b6eaa09e1fd893ce298b
Jul 17 21:51:01.147010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:01.146915    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerStarted","Data":"6c55911392670aac55453e1d5e32b2ad5c7396ce3fdbbe3a5111f1201d12abba"}
Jul 17 21:51:01.147010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:01.146975    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerStarted","Data":"12342d19e58c842bf86c84eebaaa4fd19203d6adaaa4b6eaa09e1fd893ce298b"}
Jul 17 21:51:01.162583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:01.162532    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" podStartSLOduration=1.16251239 podStartE2EDuration="1.16251239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:01.16195073 +0000 UTC m=+2037.142321218" watchObservedRunningTime="2026-07-17 21:51:01.16251239 +0000 UTC m=+2037.142882868"
Jul 17 21:51:02.153827 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:02.153795    2546 generic.go:356] "Generic (PLEG): container finished" podID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerID="6c55911392670aac55453e1d5e32b2ad5c7396ce3fdbbe3a5111f1201d12abba" exitCode=6
Jul 17 21:51:02.154303 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:02.153841    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerDied","Data":"6c55911392670aac55453e1d5e32b2ad5c7396ce3fdbbe3a5111f1201d12abba"}
Jul 17 21:51:02.154303 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:02.154106    2546 scope.go:117] "RemoveContainer" containerID="6c55911392670aac55453e1d5e32b2ad5c7396ce3fdbbe3a5111f1201d12abba"
Jul 17 21:51:03.160244 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:03.160163    2546 generic.go:356] "Generic (PLEG): container finished" podID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerID="b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce" exitCode=6
Jul 17 21:51:03.160244 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:03.160219    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerDied","Data":"b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce"}
Jul 17 21:51:03.160701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:03.160267    2546 scope.go:117] "RemoveContainer" containerID="6c55911392670aac55453e1d5e32b2ad5c7396ce3fdbbe3a5111f1201d12abba"
Jul 17 21:51:03.160701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:03.160398    2546 scope.go:117] "RemoveContainer" containerID="b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce"
Jul 17 21:51:03.160701 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:03.160617    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2_odh-ai-gateway-infra(25231378-e7ff-43b9-95eb-103b153fbbf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6"
Jul 17 21:51:04.173864 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:04.173835    2546 scope.go:117] "RemoveContainer" containerID="b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce"
Jul 17 21:51:04.174282 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:04.174032    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2_odh-ai-gateway-infra(25231378-e7ff-43b9-95eb-103b153fbbf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6"
Jul 17 21:51:07.318878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.318843    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:07.319317 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.319173    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:07.319317 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.319186    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:07.319317 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.319246    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="92a90f08-53c7-4111-8b76-ea38aa833181" containerName="revoke-keys"
Jul 17 21:51:07.319317 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.319257    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="cabe8471-1e02-48eb-a33d-28900edaccd9" containerName="revoke-keys"
Jul 17 21:51:07.322027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.322011    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:07.329925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.329667    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:07.439635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.439607    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt\" (UID: \"c1be6590-3274-4de6-a9c7-83e7bf08f966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:07.540217 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.540183    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt\" (UID: \"c1be6590-3274-4de6-a9c7-83e7bf08f966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:07.540841 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.540820    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt\" (UID: \"c1be6590-3274-4de6-a9c7-83e7bf08f966\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:07.634291 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.634232    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:07.968102 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:07.968079    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:07.970488 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:51:07.970454    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1be6590_3274_4de6_a9c7_83e7bf08f966.slice/crio-2269c700d4ec31ab5f3be16b13aff7f1bd0f6119f6e9a9e1ee00e1a9bfb47fcb WatchSource:0}: Error finding container 2269c700d4ec31ab5f3be16b13aff7f1bd0f6119f6e9a9e1ee00e1a9bfb47fcb: Status 404 returned error can't find the container with id 2269c700d4ec31ab5f3be16b13aff7f1bd0f6119f6e9a9e1ee00e1a9bfb47fcb
Jul 17 21:51:08.196955 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:08.196928    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" event={"ID":"c1be6590-3274-4de6-a9c7-83e7bf08f966","Type":"ContainerStarted","Data":"2269c700d4ec31ab5f3be16b13aff7f1bd0f6119f6e9a9e1ee00e1a9bfb47fcb"}
Jul 17 21:51:09.203797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:09.203758    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerID="ad2ac1cf3ca815916e42b12089adef395e9925742c5bb4a396cf934952fc3596" exitCode=7
Jul 17 21:51:09.203797 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:09.203800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" event={"ID":"c1be6590-3274-4de6-a9c7-83e7bf08f966","Type":"ContainerDied","Data":"ad2ac1cf3ca815916e42b12089adef395e9925742c5bb4a396cf934952fc3596"}
Jul 17 21:51:09.204330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:09.204021    2546 scope.go:117] "RemoveContainer" containerID="ad2ac1cf3ca815916e42b12089adef395e9925742c5bb4a396cf934952fc3596"
Jul 17 21:51:10.211480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:10.211375    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerID="026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec" exitCode=7
Jul 17 21:51:10.211975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:10.211473    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" event={"ID":"c1be6590-3274-4de6-a9c7-83e7bf08f966","Type":"ContainerDied","Data":"026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec"}
Jul 17 21:51:10.211975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:10.211571    2546 scope.go:117] "RemoveContainer" containerID="ad2ac1cf3ca815916e42b12089adef395e9925742c5bb4a396cf934952fc3596"
Jul 17 21:51:10.211975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:10.211858    2546 scope.go:117] "RemoveContainer" containerID="026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec"
Jul 17 21:51:10.212141 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:10.212085    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt_odh-ai-gateway-infra(c1be6590-3274-4de6-a9c7-83e7bf08f966)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966"
Jul 17 21:51:11.218524 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:11.218498    2546 scope.go:117] "RemoveContainer" containerID="026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec"
Jul 17 21:51:11.218922 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:11.218669    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt_odh-ai-gateway-infra(c1be6590-3274-4de6-a9c7-83e7bf08f966)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966"
Jul 17 21:51:18.319744 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.319696    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:18.324199 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.324180    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:18.330653 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.330625    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:18.427819 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.427776    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw\" (UID: \"63429dee-6a47-4a0d-95c6-7bd057bad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:18.528534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.528491    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw\" (UID: \"63429dee-6a47-4a0d-95c6-7bd057bad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:18.529175 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.529151    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw\" (UID: \"63429dee-6a47-4a0d-95c6-7bd057bad877\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:18.636343 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.636286    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:18.769639 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:18.769621    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:19.260615 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:19.260582    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerStarted","Data":"2509d156510527307af68a7ae165dd1006b6a86f7627aac6df724177f9455c7b"}
Jul 17 21:51:19.260767 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:19.260635    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerStarted","Data":"a47e56b9bb7e10211437c2f6b97f42bb78b727a3f4f9254114c7f469906fe295"}
Jul 17 21:51:19.278458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:19.278412    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" podStartSLOduration=1.278397187 podStartE2EDuration="1.278397187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:51:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:51:19.277557244 +0000 UTC m=+2055.257927721" watchObservedRunningTime="2026-07-17 21:51:19.278397187 +0000 UTC m=+2055.258767664"
Jul 17 21:51:19.531139 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:19.531065    2546 scope.go:117] "RemoveContainer" containerID="b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce"
Jul 17 21:51:20.266983 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.266949    2546 generic.go:356] "Generic (PLEG): container finished" podID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerID="2509d156510527307af68a7ae165dd1006b6a86f7627aac6df724177f9455c7b" exitCode=6
Jul 17 21:51:20.267099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.267035    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerDied","Data":"2509d156510527307af68a7ae165dd1006b6a86f7627aac6df724177f9455c7b"}
Jul 17 21:51:20.267203 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.267187    2546 scope.go:117] "RemoveContainer" containerID="2509d156510527307af68a7ae165dd1006b6a86f7627aac6df724177f9455c7b"
Jul 17 21:51:20.268877 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.268855    2546 generic.go:356] "Generic (PLEG): container finished" podID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerID="fa91f0a72ec0fd48f1d7766d0ba83a78a053bc0afa68fdbdbde1701672c2119a" exitCode=6
Jul 17 21:51:20.268962 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.268923    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerDied","Data":"fa91f0a72ec0fd48f1d7766d0ba83a78a053bc0afa68fdbdbde1701672c2119a"}
Jul 17 21:51:20.269006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.268968    2546 scope.go:117] "RemoveContainer" containerID="b93ad8254591e10d60ccad27012c545115f8abdfbc2b26918b5891712c40b5ce"
Jul 17 21:51:20.269295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:20.269248    2546 scope.go:117] "RemoveContainer" containerID="fa91f0a72ec0fd48f1d7766d0ba83a78a053bc0afa68fdbdbde1701672c2119a"
Jul 17 21:51:20.269428 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:20.269408    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2_odh-ai-gateway-infra(25231378-e7ff-43b9-95eb-103b153fbbf6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6"
Jul 17 21:51:21.275526 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.275489    2546 generic.go:356] "Generic (PLEG): container finished" podID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerID="9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29" exitCode=6
Jul 17 21:51:21.275996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.275535    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerDied","Data":"9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29"}
Jul 17 21:51:21.275996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.275604    2546 scope.go:117] "RemoveContainer" containerID="2509d156510527307af68a7ae165dd1006b6a86f7627aac6df724177f9455c7b"
Jul 17 21:51:21.275996 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.275799    2546 scope.go:117] "RemoveContainer" containerID="9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29"
Jul 17 21:51:21.276186 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:21.276023    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw_odh-ai-gateway-infra(63429dee-6a47-4a0d-95c6-7bd057bad877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877"
Jul 17 21:51:21.308939 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.308917    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:21.439636 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.439616    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:21.549369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.549286    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca\") pod \"25231378-e7ff-43b9-95eb-103b153fbbf6\" (UID: \"25231378-e7ff-43b9-95eb-103b153fbbf6\") "
Jul 17 21:51:21.549616 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.549593    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25231378-e7ff-43b9-95eb-103b153fbbf6" (UID: "25231378-e7ff-43b9-95eb-103b153fbbf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:21.650193 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:21.650150    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25231378-e7ff-43b9-95eb-103b153fbbf6-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:51:22.285259 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.285223    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"
Jul 17 21:51:22.285259 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.285251    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2" event={"ID":"25231378-e7ff-43b9-95eb-103b153fbbf6","Type":"ContainerDied","Data":"12342d19e58c842bf86c84eebaaa4fd19203d6adaaa4b6eaa09e1fd893ce298b"}
Jul 17 21:51:22.285893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.285320    2546 scope.go:117] "RemoveContainer" containerID="fa91f0a72ec0fd48f1d7766d0ba83a78a053bc0afa68fdbdbde1701672c2119a"
Jul 17 21:51:22.287622 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.287590    2546 scope.go:117] "RemoveContainer" containerID="9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29"
Jul 17 21:51:22.287903 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:22.287869    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw_odh-ai-gateway-infra(63429dee-6a47-4a0d-95c6-7bd057bad877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877"
Jul 17 21:51:22.314580 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.314557    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:22.320742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.320708    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-pgln2"]
Jul 17 21:51:22.535986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:22.535921    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" path="/var/lib/kubelet/pods/25231378-e7ff-43b9-95eb-103b153fbbf6/volumes"
Jul 17 21:51:23.530991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:23.530964    2546 scope.go:117] "RemoveContainer" containerID="026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec"
Jul 17 21:51:24.300727 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:24.300695    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerID="066ec5e96f732814bdf1efff6a71a464044773aa46b35f7070bed57671ac9393" exitCode=7
Jul 17 21:51:24.300855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:24.300770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" event={"ID":"c1be6590-3274-4de6-a9c7-83e7bf08f966","Type":"ContainerDied","Data":"066ec5e96f732814bdf1efff6a71a464044773aa46b35f7070bed57671ac9393"}
Jul 17 21:51:24.300855 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:24.300834    2546 scope.go:117] "RemoveContainer" containerID="026410657ebb8a85ac34c3c2e333e74e3fe563f3fd849b581424570f9da71eec"
Jul 17 21:51:24.301123 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:24.301104    2546 scope.go:117] "RemoveContainer" containerID="066ec5e96f732814bdf1efff6a71a464044773aa46b35f7070bed57671ac9393"
Jul 17 21:51:24.301333 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:24.301312    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt_odh-ai-gateway-infra(c1be6590-3274-4de6-a9c7-83e7bf08f966)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966"
Jul 17 21:51:25.327610 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:25.327558    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:25.455790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:25.455766    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:25.581603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:25.581532    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca\") pod \"c1be6590-3274-4de6-a9c7-83e7bf08f966\" (UID: \"c1be6590-3274-4de6-a9c7-83e7bf08f966\") "
Jul 17 21:51:25.581873 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:25.581853    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1be6590-3274-4de6-a9c7-83e7bf08f966" (UID: "c1be6590-3274-4de6-a9c7-83e7bf08f966"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:25.682910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:25.682875    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1be6590-3274-4de6-a9c7-83e7bf08f966-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:51:26.314671 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.314631    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt" event={"ID":"c1be6590-3274-4de6-a9c7-83e7bf08f966","Type":"ContainerDied","Data":"2269c700d4ec31ab5f3be16b13aff7f1bd0f6119f6e9a9e1ee00e1a9bfb47fcb"}
Jul 17 21:51:26.314671 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.314670    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"
Jul 17 21:51:26.314929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.314704    2546 scope.go:117] "RemoveContainer" containerID="066ec5e96f732814bdf1efff6a71a464044773aa46b35f7070bed57671ac9393"
Jul 17 21:51:26.340218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.340187    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:26.351029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.351001    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-k4nxt"]
Jul 17 21:51:26.536837 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:26.536808    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" path="/var/lib/kubelet/pods/c1be6590-3274-4de6-a9c7-83e7bf08f966/volumes"
Jul 17 21:51:32.531475 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:32.531442    2546 scope.go:117] "RemoveContainer" containerID="9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29"
Jul 17 21:51:33.353197 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:33.353162    2546 generic.go:356] "Generic (PLEG): container finished" podID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerID="1a2df105b4d8aa7122bdf85c1c01f4bf142d7ba2fcb5e56d80aba1c5dafe5aa7" exitCode=6
Jul 17 21:51:33.353322 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:33.353233    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerDied","Data":"1a2df105b4d8aa7122bdf85c1c01f4bf142d7ba2fcb5e56d80aba1c5dafe5aa7"}
Jul 17 21:51:33.353322 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:33.353301    2546 scope.go:117] "RemoveContainer" containerID="9b2a2f6281d3eee7af7971c6d1ad93ba2065eed0a99a80f019fee22c26504c29"
Jul 17 21:51:33.353609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:33.353575    2546 scope.go:117] "RemoveContainer" containerID="1a2df105b4d8aa7122bdf85c1c01f4bf142d7ba2fcb5e56d80aba1c5dafe5aa7"
Jul 17 21:51:33.353870 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:33.353848    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw_odh-ai-gateway-infra(63429dee-6a47-4a0d-95c6-7bd057bad877)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877"
Jul 17 21:51:34.384193 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:34.384161    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:34.515686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:34.515664    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:34.652596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:34.652510    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca\") pod \"63429dee-6a47-4a0d-95c6-7bd057bad877\" (UID: \"63429dee-6a47-4a0d-95c6-7bd057bad877\") "
Jul 17 21:51:34.652901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:34.652881    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63429dee-6a47-4a0d-95c6-7bd057bad877" (UID: "63429dee-6a47-4a0d-95c6-7bd057bad877"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:51:34.753912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:34.753869    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63429dee-6a47-4a0d-95c6-7bd057bad877-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:51:35.373929 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:35.373896    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"
Jul 17 21:51:35.374108 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:35.373893    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw" event={"ID":"63429dee-6a47-4a0d-95c6-7bd057bad877","Type":"ContainerDied","Data":"a47e56b9bb7e10211437c2f6b97f42bb78b727a3f4f9254114c7f469906fe295"}
Jul 17 21:51:35.374108 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:35.374067    2546 scope.go:117] "RemoveContainer" containerID="1a2df105b4d8aa7122bdf85c1c01f4bf142d7ba2fcb5e56d80aba1c5dafe5aa7"
Jul 17 21:51:35.396950 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:35.396922    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:35.403102 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:35.403078    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-54nbw"]
Jul 17 21:51:36.536742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:36.536699    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" path="/var/lib/kubelet/pods/63429dee-6a47-4a0d-95c6-7bd057bad877/volumes"
Jul 17 21:51:57.361392 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361357    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361636    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361647    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361658    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361664    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361669    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361675    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361684    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361689    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361694    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361699    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361708    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361713    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361735    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361740    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.361791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361795    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361803    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361808    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361814    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361819    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361825    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:51:57.362229 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.361831    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:51:57.364773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.364754    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:51:57.367052 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.367033    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:51:57.371650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.371623    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:51:57.417033 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.417008    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs\" (UID: \"7fd274d2-565a-47e6-951b-5b0c7301d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:51:57.517362 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.517342    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs\" (UID: \"7fd274d2-565a-47e6-951b-5b0c7301d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:51:57.517973 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.517951    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs\" (UID: \"7fd274d2-565a-47e6-951b-5b0c7301d576\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:51:57.675900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.675840    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:51:57.803176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:57.803153    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:51:57.805918 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:51:57.805886    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fd274d2_565a_47e6_951b_5b0c7301d576.slice/crio-f4c7cd36c453cf2fa4489a06ffde35389f5eeb5f8d24720c2c554b47fdd4e9cd WatchSource:0}: Error finding container f4c7cd36c453cf2fa4489a06ffde35389f5eeb5f8d24720c2c554b47fdd4e9cd: Status 404 returned error can't find the container with id f4c7cd36c453cf2fa4489a06ffde35389f5eeb5f8d24720c2c554b47fdd4e9cd
Jul 17 21:51:58.488235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:58.488195    2546 generic.go:356] "Generic (PLEG): container finished" podID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerID="a8dbebd8859075fbd3efef6283f2498f54cf65fce767ea66b0e7fad28d5dc18d" exitCode=7
Jul 17 21:51:58.488596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:58.488278    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" event={"ID":"7fd274d2-565a-47e6-951b-5b0c7301d576","Type":"ContainerDied","Data":"a8dbebd8859075fbd3efef6283f2498f54cf65fce767ea66b0e7fad28d5dc18d"}
Jul 17 21:51:58.488596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:58.488331    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" event={"ID":"7fd274d2-565a-47e6-951b-5b0c7301d576","Type":"ContainerStarted","Data":"f4c7cd36c453cf2fa4489a06ffde35389f5eeb5f8d24720c2c554b47fdd4e9cd"}
Jul 17 21:51:58.488596 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:58.488467    2546 scope.go:117] "RemoveContainer" containerID="a8dbebd8859075fbd3efef6283f2498f54cf65fce767ea66b0e7fad28d5dc18d"
Jul 17 21:51:59.494454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:59.494420    2546 generic.go:356] "Generic (PLEG): container finished" podID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerID="455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee" exitCode=7
Jul 17 21:51:59.494918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:59.494514    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" event={"ID":"7fd274d2-565a-47e6-951b-5b0c7301d576","Type":"ContainerDied","Data":"455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee"}
Jul 17 21:51:59.494918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:59.494603    2546 scope.go:117] "RemoveContainer" containerID="a8dbebd8859075fbd3efef6283f2498f54cf65fce767ea66b0e7fad28d5dc18d"
Jul 17 21:51:59.494918 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:51:59.494665    2546 scope.go:117] "RemoveContainer" containerID="455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee"
Jul 17 21:51:59.494918 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:51:59.494892    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs_odh-ai-gateway-infra(7fd274d2-565a-47e6-951b-5b0c7301d576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576"
Jul 17 21:52:00.196087 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196054    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:00.196347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196335    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:52:00.196397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196348    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="63429dee-6a47-4a0d-95c6-7bd057bad877" containerName="revoke-keys"
Jul 17 21:52:00.196397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196366    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:52:00.196397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196372    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:52:00.196488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196419    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1be6590-3274-4de6-a9c7-83e7bf08f966" containerName="revoke-keys"
Jul 17 21:52:00.196488 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.196428    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="25231378-e7ff-43b9-95eb-103b153fbbf6" containerName="revoke-keys"
Jul 17 21:52:00.200374 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.200358    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:00.207916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.207888    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:00.236318 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.236292    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp\" (UID: \"37d26ad3-ddd4-4376-bf25-c4d6653b222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:00.337505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.337475    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp\" (UID: \"37d26ad3-ddd4-4376-bf25-c4d6653b222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:00.338099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.338080    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp\" (UID: \"37d26ad3-ddd4-4376-bf25-c4d6653b222f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:00.500666 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.500639    2546 scope.go:117] "RemoveContainer" containerID="455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee"
Jul 17 21:52:00.501059 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:00.500842    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs_odh-ai-gateway-infra(7fd274d2-565a-47e6-951b-5b0c7301d576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576"
Jul 17 21:52:00.511822 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.511804    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:00.641391 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:00.641371    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:00.643363 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:52:00.643338    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37d26ad3_ddd4_4376_bf25_c4d6653b222f.slice/crio-0625d1c7e75f835b878cf5c84ba6fc2f39e239bac4d7cf57aeafb9c3f32853a7 WatchSource:0}: Error finding container 0625d1c7e75f835b878cf5c84ba6fc2f39e239bac4d7cf57aeafb9c3f32853a7: Status 404 returned error can't find the container with id 0625d1c7e75f835b878cf5c84ba6fc2f39e239bac4d7cf57aeafb9c3f32853a7
Jul 17 21:52:01.506163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:01.506130    2546 generic.go:356] "Generic (PLEG): container finished" podID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerID="c650c76c5d180cf8a12301ae526e3ed7859936737dcdf5cbe0cf92c92d597ffb" exitCode=6
Jul 17 21:52:01.506561 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:01.506219    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" event={"ID":"37d26ad3-ddd4-4376-bf25-c4d6653b222f","Type":"ContainerDied","Data":"c650c76c5d180cf8a12301ae526e3ed7859936737dcdf5cbe0cf92c92d597ffb"}
Jul 17 21:52:01.506657 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:01.506624    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" event={"ID":"37d26ad3-ddd4-4376-bf25-c4d6653b222f","Type":"ContainerStarted","Data":"0625d1c7e75f835b878cf5c84ba6fc2f39e239bac4d7cf57aeafb9c3f32853a7"}
Jul 17 21:52:01.510793 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:01.507017    2546 scope.go:117] "RemoveContainer" containerID="c650c76c5d180cf8a12301ae526e3ed7859936737dcdf5cbe0cf92c92d597ffb"
Jul 17 21:52:02.513798 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:02.513763    2546 generic.go:356] "Generic (PLEG): container finished" podID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerID="d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e" exitCode=6
Jul 17 21:52:02.514218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:02.513842    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" event={"ID":"37d26ad3-ddd4-4376-bf25-c4d6653b222f","Type":"ContainerDied","Data":"d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e"}
Jul 17 21:52:02.514218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:02.513900    2546 scope.go:117] "RemoveContainer" containerID="c650c76c5d180cf8a12301ae526e3ed7859936737dcdf5cbe0cf92c92d597ffb"
Jul 17 21:52:02.514218 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:02.514080    2546 scope.go:117] "RemoveContainer" containerID="d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e"
Jul 17 21:52:02.514431 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:02.514264    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp_odh-ai-gateway-infra(37d26ad3-ddd4-4376-bf25-c4d6653b222f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f"
Jul 17 21:52:03.521066 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:03.521036    2546 scope.go:117] "RemoveContainer" containerID="d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e"
Jul 17 21:52:03.521517 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:03.521234    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp_odh-ai-gateway-infra(37d26ad3-ddd4-4376-bf25-c4d6653b222f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f"
Jul 17 21:52:04.615763 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:04.615706    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:52:04.623380 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:04.623359    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:52:08.368529 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.368495    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:08.371618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.371587    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:08.379629 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.379590    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:08.506105 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.506071    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl\" (UID: \"222548f5-be77-4567-b92f-fc01bff37e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:08.606578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.606544    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl\" (UID: \"222548f5-be77-4567-b92f-fc01bff37e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:08.607236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.607211    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl\" (UID: \"222548f5-be77-4567-b92f-fc01bff37e31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:08.683990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.683926    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:08.816429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:08.816405    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:08.818357 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:52:08.818330    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222548f5_be77_4567_b92f_fc01bff37e31.slice/crio-45bfe5ea5372f45fe6b1b78e3507836c5da4db12215e324a0c42c7b5ea2332aa WatchSource:0}: Error finding container 45bfe5ea5372f45fe6b1b78e3507836c5da4db12215e324a0c42c7b5ea2332aa: Status 404 returned error can't find the container with id 45bfe5ea5372f45fe6b1b78e3507836c5da4db12215e324a0c42c7b5ea2332aa
Jul 17 21:52:09.553634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:09.553600    2546 generic.go:356] "Generic (PLEG): container finished" podID="222548f5-be77-4567-b92f-fc01bff37e31" containerID="36c5a01cb5f749dc5bf85b79dd3a23f659810883e4b2289973dc11a448cb7994" exitCode=6
Jul 17 21:52:09.554134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:09.553649    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" event={"ID":"222548f5-be77-4567-b92f-fc01bff37e31","Type":"ContainerDied","Data":"36c5a01cb5f749dc5bf85b79dd3a23f659810883e4b2289973dc11a448cb7994"}
Jul 17 21:52:09.554134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:09.553714    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" event={"ID":"222548f5-be77-4567-b92f-fc01bff37e31","Type":"ContainerStarted","Data":"45bfe5ea5372f45fe6b1b78e3507836c5da4db12215e324a0c42c7b5ea2332aa"}
Jul 17 21:52:09.554134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:09.553961    2546 scope.go:117] "RemoveContainer" containerID="36c5a01cb5f749dc5bf85b79dd3a23f659810883e4b2289973dc11a448cb7994"
Jul 17 21:52:10.560024 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:10.559981    2546 generic.go:356] "Generic (PLEG): container finished" podID="222548f5-be77-4567-b92f-fc01bff37e31" containerID="540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab" exitCode=6
Jul 17 21:52:10.560429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:10.560064    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" event={"ID":"222548f5-be77-4567-b92f-fc01bff37e31","Type":"ContainerDied","Data":"540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab"}
Jul 17 21:52:10.560429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:10.560137    2546 scope.go:117] "RemoveContainer" containerID="36c5a01cb5f749dc5bf85b79dd3a23f659810883e4b2289973dc11a448cb7994"
Jul 17 21:52:10.560429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:10.560364    2546 scope.go:117] "RemoveContainer" containerID="540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab"
Jul 17 21:52:10.560571 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:10.560536    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl_odh-ai-gateway-infra(222548f5-be77-4567-b92f-fc01bff37e31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" podUID="222548f5-be77-4567-b92f-fc01bff37e31"
Jul 17 21:52:11.531195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:11.531155    2546 scope.go:117] "RemoveContainer" containerID="455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee"
Jul 17 21:52:11.565903 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:11.565879    2546 scope.go:117] "RemoveContainer" containerID="540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab"
Jul 17 21:52:11.566190 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:11.566063    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl_odh-ai-gateway-infra(222548f5-be77-4567-b92f-fc01bff37e31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" podUID="222548f5-be77-4567-b92f-fc01bff37e31"
Jul 17 21:52:12.572061 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:12.572030    2546 generic.go:356] "Generic (PLEG): container finished" podID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerID="343b7eef2d4cc9295e62ab3f2b52c975300ba60a3e7ac43d9e61961d28ac45eb" exitCode=7
Jul 17 21:52:12.572480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:12.572101    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" event={"ID":"7fd274d2-565a-47e6-951b-5b0c7301d576","Type":"ContainerDied","Data":"343b7eef2d4cc9295e62ab3f2b52c975300ba60a3e7ac43d9e61961d28ac45eb"}
Jul 17 21:52:12.572480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:12.572151    2546 scope.go:117] "RemoveContainer" containerID="455e9099a38ac98875a23a5aaad685137f2cd3513905d94d2aef07e6d805a7ee"
Jul 17 21:52:12.572480 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:12.572397    2546 scope.go:117] "RemoveContainer" containerID="343b7eef2d4cc9295e62ab3f2b52c975300ba60a3e7ac43d9e61961d28ac45eb"
Jul 17 21:52:12.572640 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:12.572586    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs_odh-ai-gateway-infra(7fd274d2-565a-47e6-951b-5b0c7301d576)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576"
Jul 17 21:52:13.609239 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:13.609205    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:52:13.738304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:13.738283    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:52:13.849074 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:13.849050    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca\") pod \"7fd274d2-565a-47e6-951b-5b0c7301d576\" (UID: \"7fd274d2-565a-47e6-951b-5b0c7301d576\") "
Jul 17 21:52:13.849366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:13.849344    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7fd274d2-565a-47e6-951b-5b0c7301d576" (UID: "7fd274d2-565a-47e6-951b-5b0c7301d576"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:13.949804 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:13.949740    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7fd274d2-565a-47e6-951b-5b0c7301d576-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:52:14.585090 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:14.585068    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"
Jul 17 21:52:14.585270 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:14.585109    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs" event={"ID":"7fd274d2-565a-47e6-951b-5b0c7301d576","Type":"ContainerDied","Data":"f4c7cd36c453cf2fa4489a06ffde35389f5eeb5f8d24720c2c554b47fdd4e9cd"}
Jul 17 21:52:14.585270 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:14.585164    2546 scope.go:117] "RemoveContainer" containerID="343b7eef2d4cc9295e62ab3f2b52c975300ba60a3e7ac43d9e61961d28ac45eb"
Jul 17 21:52:14.601322 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:14.601300    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:52:14.607579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:14.607555    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-6pngs"]
Jul 17 21:52:16.536892 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:16.536856    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" path="/var/lib/kubelet/pods/7fd274d2-565a-47e6-951b-5b0c7301d576/volumes"
Jul 17 21:52:18.531590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:18.531560    2546 scope.go:117] "RemoveContainer" containerID="d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e"
Jul 17 21:52:19.611821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:19.611790    2546 generic.go:356] "Generic (PLEG): container finished" podID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerID="b9719071313cde5910a9b1eb2e55ab219d80ed865948d9b80c075c2960783623" exitCode=6
Jul 17 21:52:19.612230 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:19.611865    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" event={"ID":"37d26ad3-ddd4-4376-bf25-c4d6653b222f","Type":"ContainerDied","Data":"b9719071313cde5910a9b1eb2e55ab219d80ed865948d9b80c075c2960783623"}
Jul 17 21:52:19.612230 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:19.611926    2546 scope.go:117] "RemoveContainer" containerID="d6722e74e0cbabfb5cd7a78938a635c48888094c834c8158338a0f8c8f5c2d4e"
Jul 17 21:52:19.612230 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:19.612134    2546 scope.go:117] "RemoveContainer" containerID="b9719071313cde5910a9b1eb2e55ab219d80ed865948d9b80c075c2960783623"
Jul 17 21:52:19.612374 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:19.612311    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp_odh-ai-gateway-infra(37d26ad3-ddd4-4376-bf25-c4d6653b222f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f"
Jul 17 21:52:20.639384 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:20.639355    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:20.768582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:20.768561    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:20.904910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:20.904833    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca\") pod \"37d26ad3-ddd4-4376-bf25-c4d6653b222f\" (UID: \"37d26ad3-ddd4-4376-bf25-c4d6653b222f\") "
Jul 17 21:52:20.905252 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:20.905226    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "37d26ad3-ddd4-4376-bf25-c4d6653b222f" (UID: "37d26ad3-ddd4-4376-bf25-c4d6653b222f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:21.005869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.005844    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37d26ad3-ddd4-4376-bf25-c4d6653b222f-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:52:21.625697 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.625659    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"
Jul 17 21:52:21.625940 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.625658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp" event={"ID":"37d26ad3-ddd4-4376-bf25-c4d6653b222f","Type":"ContainerDied","Data":"0625d1c7e75f835b878cf5c84ba6fc2f39e239bac4d7cf57aeafb9c3f32853a7"}
Jul 17 21:52:21.625940 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.625815    2546 scope.go:117] "RemoveContainer" containerID="b9719071313cde5910a9b1eb2e55ab219d80ed865948d9b80c075c2960783623"
Jul 17 21:52:21.647481 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.647453    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:21.653514 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:21.653489    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w8qkp"]
Jul 17 21:52:22.537225 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:22.537192    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" path="/var/lib/kubelet/pods/37d26ad3-ddd4-4376-bf25-c4d6653b222f/volumes"
Jul 17 21:52:25.531109 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:25.531083    2546 scope.go:117] "RemoveContainer" containerID="540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab"
Jul 17 21:52:26.652865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:26.652831    2546 generic.go:356] "Generic (PLEG): container finished" podID="222548f5-be77-4567-b92f-fc01bff37e31" containerID="23402e88b6ec9a6b38ef15047e91ebb0ae9968ba8ec2ee24bf33fb1af8eeda3c" exitCode=6
Jul 17 21:52:26.653310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:26.652897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" event={"ID":"222548f5-be77-4567-b92f-fc01bff37e31","Type":"ContainerDied","Data":"23402e88b6ec9a6b38ef15047e91ebb0ae9968ba8ec2ee24bf33fb1af8eeda3c"}
Jul 17 21:52:26.653310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:26.652955    2546 scope.go:117] "RemoveContainer" containerID="540160d3ead44c0c0f5e531738ed9e0e47910de4a817019dfa7b5bedaf45c8ab"
Jul 17 21:52:26.653310 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:26.653273    2546 scope.go:117] "RemoveContainer" containerID="23402e88b6ec9a6b38ef15047e91ebb0ae9968ba8ec2ee24bf33fb1af8eeda3c"
Jul 17 21:52:26.653472 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:26.653449    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl_odh-ai-gateway-infra(222548f5-be77-4567-b92f-fc01bff37e31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" podUID="222548f5-be77-4567-b92f-fc01bff37e31"
Jul 17 21:52:27.679054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:27.679021    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:27.809132 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:27.809109    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:27.961299 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:27.961227    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca\") pod \"222548f5-be77-4567-b92f-fc01bff37e31\" (UID: \"222548f5-be77-4567-b92f-fc01bff37e31\") "
Jul 17 21:52:27.961584 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:27.961561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "222548f5-be77-4567-b92f-fc01bff37e31" (UID: "222548f5-be77-4567-b92f-fc01bff37e31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:52:28.062493 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.062463    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/222548f5-be77-4567-b92f-fc01bff37e31-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:52:28.671450 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.671425    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"
Jul 17 21:52:28.671607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.671434    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl" event={"ID":"222548f5-be77-4567-b92f-fc01bff37e31","Type":"ContainerDied","Data":"45bfe5ea5372f45fe6b1b78e3507836c5da4db12215e324a0c42c7b5ea2332aa"}
Jul 17 21:52:28.671607 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.671521    2546 scope.go:117] "RemoveContainer" containerID="23402e88b6ec9a6b38ef15047e91ebb0ae9968ba8ec2ee24bf33fb1af8eeda3c"
Jul 17 21:52:28.690354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.690327    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:28.697014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:28.696992    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-zgpvl"]
Jul 17 21:52:30.536641 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:30.536601    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="222548f5-be77-4567-b92f-fc01bff37e31" path="/var/lib/kubelet/pods/222548f5-be77-4567-b92f-fc01bff37e31/volumes"
Jul 17 21:52:47.401818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.401781    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402091    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402103    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402114    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402123    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402130    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402136    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402160    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402165    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402174    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402178    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402187    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402196    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402204    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402211    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402220    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402225    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402231    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402236    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402292    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402304    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402311    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402315    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402321    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fd274d2-565a-47e6-951b-5b0c7301d576" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402325    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:52:47.402817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.402330    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:52:47.405099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.405082    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:52:47.407355 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.407328    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:52:47.407547 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.407530    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh\" (UID: \"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:52:47.412412 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.412388    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:52:47.508331 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.508297    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh\" (UID: \"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:52:47.508876 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.508859    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh\" (UID: \"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:52:47.716786 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.716696    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:52:47.850543 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:47.850520    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:52:47.852753 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:52:47.852714    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90ae3608_33ba_4f22_95f6_c9ccfb0a2da4.slice/crio-e9a736a7c428e475b1be783cd333d0413ca179d5fa959660b3f61de991bbecc1 WatchSource:0}: Error finding container e9a736a7c428e475b1be783cd333d0413ca179d5fa959660b3f61de991bbecc1: Status 404 returned error can't find the container with id e9a736a7c428e475b1be783cd333d0413ca179d5fa959660b3f61de991bbecc1
Jul 17 21:52:48.778736 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:48.778682    2546 generic.go:356] "Generic (PLEG): container finished" podID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerID="2268251d236b26c2ed917f93d48a5e6c5ea3ef05d423d3842895536050445ac9" exitCode=7
Jul 17 21:52:48.779159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:48.778776    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerDied","Data":"2268251d236b26c2ed917f93d48a5e6c5ea3ef05d423d3842895536050445ac9"}
Jul 17 21:52:48.779159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:48.778828    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerStarted","Data":"e9a736a7c428e475b1be783cd333d0413ca179d5fa959660b3f61de991bbecc1"}
Jul 17 21:52:48.779159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:48.778919    2546 scope.go:117] "RemoveContainer" containerID="2268251d236b26c2ed917f93d48a5e6c5ea3ef05d423d3842895536050445ac9"
Jul 17 21:52:49.786299 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:49.786264    2546 generic.go:356] "Generic (PLEG): container finished" podID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerID="0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b" exitCode=7
Jul 17 21:52:49.786701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:49.786345    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerDied","Data":"0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b"}
Jul 17 21:52:49.786701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:49.786401    2546 scope.go:117] "RemoveContainer" containerID="2268251d236b26c2ed917f93d48a5e6c5ea3ef05d423d3842895536050445ac9"
Jul 17 21:52:49.786701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:49.786577    2546 scope.go:117] "RemoveContainer" containerID="0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b"
Jul 17 21:52:49.786926 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:49.786799    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh_odh-ai-gateway-infra(90ae3608-33ba-4f22-95f6-c9ccfb0a2da4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4"
Jul 17 21:52:50.792776 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:52:50.792748    2546 scope.go:117] "RemoveContainer" containerID="0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b"
Jul 17 21:52:50.793180 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:52:50.792939    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh_odh-ai-gateway-infra(90ae3608-33ba-4f22-95f6-c9ccfb0a2da4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4"
Jul 17 21:53:00.246614 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.246528    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:00.247010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.246898    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="222548f5-be77-4567-b92f-fc01bff37e31" containerName="revoke-keys"
Jul 17 21:53:00.247010 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.246911    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="37d26ad3-ddd4-4376-bf25-c4d6653b222f" containerName="revoke-keys"
Jul 17 21:53:00.251056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.251039    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:00.257227 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.257135    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:00.303756 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.303702    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl\" (UID: \"110e290d-b12b-43ee-802e-3cd192436e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:00.404491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.404462    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl\" (UID: \"110e290d-b12b-43ee-802e-3cd192436e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:00.405044 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.405026    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl\" (UID: \"110e290d-b12b-43ee-802e-3cd192436e98\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:00.563937 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.563869    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:00.695805 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.695779    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:00.845144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:00.845062    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerStarted","Data":"b6107ccca5b80c183134b45a9a8fbf1f229690e8f803a82c49e303ebc7385279"}
Jul 17 21:53:01.852313 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:01.852274    2546 generic.go:356] "Generic (PLEG): container finished" podID="110e290d-b12b-43ee-802e-3cd192436e98" containerID="e02afca7d9fa2c61c1ef73756793e701840207032a629976a391615e921dfdc5" exitCode=6
Jul 17 21:53:01.852710 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:01.852369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerDied","Data":"e02afca7d9fa2c61c1ef73756793e701840207032a629976a391615e921dfdc5"}
Jul 17 21:53:01.852710 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:01.852538    2546 scope.go:117] "RemoveContainer" containerID="e02afca7d9fa2c61c1ef73756793e701840207032a629976a391615e921dfdc5"
Jul 17 21:53:02.859304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:02.859270    2546 generic.go:356] "Generic (PLEG): container finished" podID="110e290d-b12b-43ee-802e-3cd192436e98" containerID="0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242" exitCode=6
Jul 17 21:53:02.859709 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:02.859353    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerDied","Data":"0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242"}
Jul 17 21:53:02.859709 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:02.859412    2546 scope.go:117] "RemoveContainer" containerID="e02afca7d9fa2c61c1ef73756793e701840207032a629976a391615e921dfdc5"
Jul 17 21:53:02.859709 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:02.859514    2546 scope.go:117] "RemoveContainer" containerID="0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242"
Jul 17 21:53:02.859709 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:02.859699    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl_odh-ai-gateway-infra(110e290d-b12b-43ee-802e-3cd192436e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" podUID="110e290d-b12b-43ee-802e-3cd192436e98"
Jul 17 21:53:03.866590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:03.866559    2546 scope.go:117] "RemoveContainer" containerID="0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242"
Jul 17 21:53:03.867037 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:03.866754    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl_odh-ai-gateway-infra(110e290d-b12b-43ee-802e-3cd192436e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" podUID="110e290d-b12b-43ee-802e-3cd192436e98"
Jul 17 21:53:05.531646 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:05.531619    2546 scope.go:117] "RemoveContainer" containerID="0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b"
Jul 17 21:53:05.878169 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:05.878146    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerStarted","Data":"eeed1e60612a8f9a86217dcae96391a450754b59410c1eb1dac51627105aa215"}
Jul 17 21:53:05.895082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:05.895038    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" podStartSLOduration=18.895025358 podStartE2EDuration="18.895025358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:52:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:05.892887469 +0000 UTC m=+2161.873257956" watchObservedRunningTime="2026-07-17 21:53:05.895025358 +0000 UTC m=+2161.875395835"
Jul 17 21:53:06.885316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:06.885277    2546 generic.go:356] "Generic (PLEG): container finished" podID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerID="eeed1e60612a8f9a86217dcae96391a450754b59410c1eb1dac51627105aa215" exitCode=7
Jul 17 21:53:06.885811 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:06.885354    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerDied","Data":"eeed1e60612a8f9a86217dcae96391a450754b59410c1eb1dac51627105aa215"}
Jul 17 21:53:06.885811 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:06.885416    2546 scope.go:117] "RemoveContainer" containerID="0bde418abcf24198a3220ce5ee49935fb077758923dac003e3adc2630f162c9b"
Jul 17 21:53:06.885811 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:06.885685    2546 scope.go:117] "RemoveContainer" containerID="eeed1e60612a8f9a86217dcae96391a450754b59410c1eb1dac51627105aa215"
Jul 17 21:53:06.886003 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:06.885916    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh_odh-ai-gateway-infra(90ae3608-33ba-4f22-95f6-c9ccfb0a2da4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4"
Jul 17 21:53:06.903072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:06.903046    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:53:08.022624 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.022599    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:53:08.064420 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.064399    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca\") pod \"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4\" (UID: \"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4\") "
Jul 17 21:53:08.068745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.065155    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" (UID: "90ae3608-33ba-4f22-95f6-c9ccfb0a2da4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:08.166136 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.166079    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:53:08.431698 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.431628    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:08.432022 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432005    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.432069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432024    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.432069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432039    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.432069 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432045    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.432159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432109    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.432159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.432119    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:08.435004 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.434985    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:08.442964 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.442936    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:08.469001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.468978    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn\" (UID: \"951943c6-39e3-4bef-b491-3b8b8c972b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:08.569484 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.569459    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn\" (UID: \"951943c6-39e3-4bef-b491-3b8b8c972b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:08.570099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.570081    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn\" (UID: \"951943c6-39e3-4bef-b491-3b8b8c972b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:08.747612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.747569    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:08.879221 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.879201    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:08.881827 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:53:08.881803    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951943c6_39e3_4bef_b491_3b8b8c972b62.slice/crio-45ec0041883e3042be03acfa0a7be4a2ca7c94685d0d3470a59b4363c95c1841 WatchSource:0}: Error finding container 45ec0041883e3042be03acfa0a7be4a2ca7c94685d0d3470a59b4363c95c1841: Status 404 returned error can't find the container with id 45ec0041883e3042be03acfa0a7be4a2ca7c94685d0d3470a59b4363c95c1841
Jul 17 21:53:08.897955 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.897876    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerStarted","Data":"45ec0041883e3042be03acfa0a7be4a2ca7c94685d0d3470a59b4363c95c1841"}
Jul 17 21:53:08.899125 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.899104    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh" event={"ID":"90ae3608-33ba-4f22-95f6-c9ccfb0a2da4","Type":"ContainerDied","Data":"e9a736a7c428e475b1be783cd333d0413ca179d5fa959660b3f61de991bbecc1"}
Jul 17 21:53:08.899232 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.899131    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"
Jul 17 21:53:08.899232 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.899162    2546 scope.go:117] "RemoveContainer" containerID="eeed1e60612a8f9a86217dcae96391a450754b59410c1eb1dac51627105aa215"
Jul 17 21:53:08.923886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.923861    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:53:08.927613 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:08.927583    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4cwdh"]
Jul 17 21:53:09.906955 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:09.906926    2546 generic.go:356] "Generic (PLEG): container finished" podID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerID="0ddd4df610c7fa1ce20ac4f6e7298e42637ba091429651d919cedf5f08a8214d" exitCode=6
Jul 17 21:53:09.907352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:09.906996    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerDied","Data":"0ddd4df610c7fa1ce20ac4f6e7298e42637ba091429651d919cedf5f08a8214d"}
Jul 17 21:53:09.907352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:09.907184    2546 scope.go:117] "RemoveContainer" containerID="0ddd4df610c7fa1ce20ac4f6e7298e42637ba091429651d919cedf5f08a8214d"
Jul 17 21:53:10.536656 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:10.536632    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" path="/var/lib/kubelet/pods/90ae3608-33ba-4f22-95f6-c9ccfb0a2da4/volumes"
Jul 17 21:53:10.913587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:10.913559    2546 generic.go:356] "Generic (PLEG): container finished" podID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerID="6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b" exitCode=6
Jul 17 21:53:10.914030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:10.913642    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerDied","Data":"6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b"}
Jul 17 21:53:10.914030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:10.913700    2546 scope.go:117] "RemoveContainer" containerID="0ddd4df610c7fa1ce20ac4f6e7298e42637ba091429651d919cedf5f08a8214d"
Jul 17 21:53:10.914030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:10.913871    2546 scope.go:117] "RemoveContainer" containerID="6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b"
Jul 17 21:53:10.914175 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:10.914092    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn_odh-ai-gateway-infra(951943c6-39e3-4bef-b491-3b8b8c972b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62"
Jul 17 21:53:11.920981 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:11.920947    2546 scope.go:117] "RemoveContainer" containerID="6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b"
Jul 17 21:53:11.921398 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:11.921155    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn_odh-ai-gateway-infra(951943c6-39e3-4bef-b491-3b8b8c972b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62"
Jul 17 21:53:19.531618 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:19.531578    2546 scope.go:117] "RemoveContainer" containerID="0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242"
Jul 17 21:53:19.969124 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:19.969090    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerStarted","Data":"92796e56013525ac711f49ea07a4b72d663bd34c3eeb4a56323e3d6f941bae7e"}
Jul 17 21:53:19.985820 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:19.985776    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" podStartSLOduration=19.985762944 podStartE2EDuration="19.985762944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:19.984387926 +0000 UTC m=+2175.964758404" watchObservedRunningTime="2026-07-17 21:53:19.985762944 +0000 UTC m=+2175.966133421"
Jul 17 21:53:20.975540 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:20.975506    2546 generic.go:356] "Generic (PLEG): container finished" podID="110e290d-b12b-43ee-802e-3cd192436e98" containerID="92796e56013525ac711f49ea07a4b72d663bd34c3eeb4a56323e3d6f941bae7e" exitCode=6
Jul 17 21:53:20.975540 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:20.975542    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerDied","Data":"92796e56013525ac711f49ea07a4b72d663bd34c3eeb4a56323e3d6f941bae7e"}
Jul 17 21:53:20.976014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:20.975606    2546 scope.go:117] "RemoveContainer" containerID="0f2626057d59ed25cb6dccd27889e1caa230e66b7bfe88bdb9e5b185e4bc5242"
Jul 17 21:53:20.976014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:20.975914    2546 scope.go:117] "RemoveContainer" containerID="92796e56013525ac711f49ea07a4b72d663bd34c3eeb4a56323e3d6f941bae7e"
Jul 17 21:53:20.976121 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:20.976099    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl_odh-ai-gateway-infra(110e290d-b12b-43ee-802e-3cd192436e98)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" podUID="110e290d-b12b-43ee-802e-3cd192436e98"
Jul 17 21:53:21.001924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:21.001899    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:22.114278 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.114255    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:22.169468 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.169439    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca\") pod \"110e290d-b12b-43ee-802e-3cd192436e98\" (UID: \"110e290d-b12b-43ee-802e-3cd192436e98\") "
Jul 17 21:53:22.169814 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.169790    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "110e290d-b12b-43ee-802e-3cd192436e98" (UID: "110e290d-b12b-43ee-802e-3cd192436e98"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:22.270508 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.270469    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/110e290d-b12b-43ee-802e-3cd192436e98-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:53:22.994358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.994323    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl" event={"ID":"110e290d-b12b-43ee-802e-3cd192436e98","Type":"ContainerDied","Data":"b6107ccca5b80c183134b45a9a8fbf1f229690e8f803a82c49e303ebc7385279"}
Jul 17 21:53:22.994358 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.994354    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"
Jul 17 21:53:22.994661 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:22.994410    2546 scope.go:117] "RemoveContainer" containerID="92796e56013525ac711f49ea07a4b72d663bd34c3eeb4a56323e3d6f941bae7e"
Jul 17 21:53:23.011572 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:23.011547    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:23.016445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:23.016426    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9lxtl"]
Jul 17 21:53:24.536704 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:24.536671    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="110e290d-b12b-43ee-802e-3cd192436e98" path="/var/lib/kubelet/pods/110e290d-b12b-43ee-802e-3cd192436e98/volumes"
Jul 17 21:53:25.531891 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:25.531862    2546 scope.go:117] "RemoveContainer" containerID="6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b"
Jul 17 21:53:26.012983 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:26.012949    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerStarted","Data":"040f1ac950641ff4e67320f10ee7ada3a69597d1abd7e2f000b01001c7222cd4"}
Jul 17 21:53:26.032890 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:26.032802    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" podStartSLOduration=18.032787641 podStartE2EDuration="18.032787641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:26.030911289 +0000 UTC m=+2182.011281766" watchObservedRunningTime="2026-07-17 21:53:26.032787641 +0000 UTC m=+2182.013158118"
Jul 17 21:53:27.020780 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:27.020712    2546 generic.go:356] "Generic (PLEG): container finished" podID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerID="040f1ac950641ff4e67320f10ee7ada3a69597d1abd7e2f000b01001c7222cd4" exitCode=6
Jul 17 21:53:27.021204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:27.020799    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerDied","Data":"040f1ac950641ff4e67320f10ee7ada3a69597d1abd7e2f000b01001c7222cd4"}
Jul 17 21:53:27.021204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:27.020874    2546 scope.go:117] "RemoveContainer" containerID="6bcfa533e39c81f166d9665d9782e7698db62273799f8523ff8edae98c0bf76b"
Jul 17 21:53:27.021204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:27.021203    2546 scope.go:117] "RemoveContainer" containerID="040f1ac950641ff4e67320f10ee7ada3a69597d1abd7e2f000b01001c7222cd4"
Jul 17 21:53:27.021397 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:27.021375    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn_odh-ai-gateway-infra(951943c6-39e3-4bef-b491-3b8b8c972b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62"
Jul 17 21:53:27.043886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:27.043861    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:28.163970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:28.163947    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:28.218222 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:28.218191    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca\") pod \"951943c6-39e3-4bef-b491-3b8b8c972b62\" (UID: \"951943c6-39e3-4bef-b491-3b8b8c972b62\") "
Jul 17 21:53:28.218562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:28.218543    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "951943c6-39e3-4bef-b491-3b8b8c972b62" (UID: "951943c6-39e3-4bef-b491-3b8b8c972b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:53:28.319571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:28.319496    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/951943c6-39e3-4bef-b491-3b8b8c972b62-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:53:29.035366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:29.035337    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"
Jul 17 21:53:29.035366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:29.035358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn" event={"ID":"951943c6-39e3-4bef-b491-3b8b8c972b62","Type":"ContainerDied","Data":"45ec0041883e3042be03acfa0a7be4a2ca7c94685d0d3470a59b4363c95c1841"}
Jul 17 21:53:29.035578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:29.035423    2546 scope.go:117] "RemoveContainer" containerID="040f1ac950641ff4e67320f10ee7ada3a69597d1abd7e2f000b01001c7222cd4"
Jul 17 21:53:29.053733 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:29.053694    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:29.065784 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:29.065754    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-98wgn"]
Jul 17 21:53:30.535783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:30.535750    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" path="/var/lib/kubelet/pods/951943c6-39e3-4bef-b491-3b8b8c972b62/volumes"
Jul 17 21:53:47.448077 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448043    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448358    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448368    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448378    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448383    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448391    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448396    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448405    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448410    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448415    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448419    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448425    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448443 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448430    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448475    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448482    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:53:47.448839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448488    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="90ae3608-33ba-4f22-95f6-c9ccfb0a2da4" containerName="revoke-keys"
Jul 17 21:53:47.448839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448496    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.448839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.448502    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:53:47.451228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.451213    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:53:47.453578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.453551    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:53:47.458562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.458261    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:53:47.573059 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.573034    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs\" (UID: \"11cef0ee-f8e0-4ad9-b544-950957ea523d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:53:47.674078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.674043    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs\" (UID: \"11cef0ee-f8e0-4ad9-b544-950957ea523d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:53:47.674657 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.674637    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs\" (UID: \"11cef0ee-f8e0-4ad9-b544-950957ea523d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:53:47.762413 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.762381    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:53:47.895880 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:47.895844    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:53:47.897692 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:53:47.897665    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11cef0ee_f8e0_4ad9_b544_950957ea523d.slice/crio-57ae984ee6d3ec45f72ef9563e6429ea31196610bdbde99cd31c1fbee9544bb6 WatchSource:0}: Error finding container 57ae984ee6d3ec45f72ef9563e6429ea31196610bdbde99cd31c1fbee9544bb6: Status 404 returned error can't find the container with id 57ae984ee6d3ec45f72ef9563e6429ea31196610bdbde99cd31c1fbee9544bb6
Jul 17 21:53:48.130754 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:48.130712    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerStarted","Data":"3e043b2b7ac702ae7ee1708e6900beaee01e371a260c186034c476412493ba4e"}
Jul 17 21:53:48.130849 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:48.130775    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerStarted","Data":"57ae984ee6d3ec45f72ef9563e6429ea31196610bdbde99cd31c1fbee9544bb6"}
Jul 17 21:53:48.151118 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:48.151071    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" podStartSLOduration=1.151055932 podStartE2EDuration="1.151055932s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:53:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:53:48.149419643 +0000 UTC m=+2204.129790121" watchObservedRunningTime="2026-07-17 21:53:48.151055932 +0000 UTC m=+2204.131426408"
Jul 17 21:53:49.138370 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:49.138323    2546 generic.go:356] "Generic (PLEG): container finished" podID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerID="3e043b2b7ac702ae7ee1708e6900beaee01e371a260c186034c476412493ba4e" exitCode=7
Jul 17 21:53:49.138872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:49.138418    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerDied","Data":"3e043b2b7ac702ae7ee1708e6900beaee01e371a260c186034c476412493ba4e"}
Jul 17 21:53:49.138872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:49.138601    2546 scope.go:117] "RemoveContainer" containerID="3e043b2b7ac702ae7ee1708e6900beaee01e371a260c186034c476412493ba4e"
Jul 17 21:53:50.145517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:50.145429    2546 generic.go:356] "Generic (PLEG): container finished" podID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerID="eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad" exitCode=7
Jul 17 21:53:50.145517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:50.145467    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerDied","Data":"eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad"}
Jul 17 21:53:50.145517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:50.145507    2546 scope.go:117] "RemoveContainer" containerID="3e043b2b7ac702ae7ee1708e6900beaee01e371a260c186034c476412493ba4e"
Jul 17 21:53:50.146179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:50.145708    2546 scope.go:117] "RemoveContainer" containerID="eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad"
Jul 17 21:53:50.146179 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:50.145910    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs_odh-ai-gateway-infra(11cef0ee-f8e0-4ad9-b544-950957ea523d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d"
Jul 17 21:53:51.151447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:53:51.151416    2546 scope.go:117] "RemoveContainer" containerID="eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad"
Jul 17 21:53:51.151932 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:53:51.151591    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs_odh-ai-gateway-infra(11cef0ee-f8e0-4ad9-b544-950957ea523d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d"
Jul 17 21:54:00.292922 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.292884    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:00.293352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.293186    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:54:00.293352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.293197    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:54:00.293352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.293259    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="110e290d-b12b-43ee-802e-3cd192436e98" containerName="revoke-keys"
Jul 17 21:54:00.293352 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.293267    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="951943c6-39e3-4bef-b491-3b8b8c972b62" containerName="revoke-keys"
Jul 17 21:54:00.296240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.296219    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:00.304389 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.304349    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:00.365975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.365954    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd\" (UID: \"1f592f89-4656-425b-a1e1-da2e3ecff7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:00.466308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.466278    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd\" (UID: \"1f592f89-4656-425b-a1e1-da2e3ecff7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:00.466875 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.466856    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd\" (UID: \"1f592f89-4656-425b-a1e1-da2e3ecff7a4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:00.609061 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.608998    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:00.740130 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:00.740105    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:00.741762 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:00.741710    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f592f89_4656_425b_a1e1_da2e3ecff7a4.slice/crio-1093803789e73a79d0b85b34c09db7bdb14f798c0193100b66be4cb13ba15e32 WatchSource:0}: Error finding container 1093803789e73a79d0b85b34c09db7bdb14f798c0193100b66be4cb13ba15e32: Status 404 returned error can't find the container with id 1093803789e73a79d0b85b34c09db7bdb14f798c0193100b66be4cb13ba15e32
Jul 17 21:54:01.206577 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:01.206546    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerStarted","Data":"19e21b1302b590c3da33c80d276973f9d9580b46f47d7d44037736c4e4dc36c8"}
Jul 17 21:54:01.206738 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:01.206629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerStarted","Data":"1093803789e73a79d0b85b34c09db7bdb14f798c0193100b66be4cb13ba15e32"}
Jul 17 21:54:01.224055 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:01.224014    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" podStartSLOduration=1.224001138 podStartE2EDuration="1.224001138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:01.221878841 +0000 UTC m=+2217.202249317" watchObservedRunningTime="2026-07-17 21:54:01.224001138 +0000 UTC m=+2217.204371616"
Jul 17 21:54:02.215515 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:02.215471    2546 generic.go:356] "Generic (PLEG): container finished" podID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerID="19e21b1302b590c3da33c80d276973f9d9580b46f47d7d44037736c4e4dc36c8" exitCode=6
Jul 17 21:54:02.216062 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:02.215549    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerDied","Data":"19e21b1302b590c3da33c80d276973f9d9580b46f47d7d44037736c4e4dc36c8"}
Jul 17 21:54:02.216062 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:02.215754    2546 scope.go:117] "RemoveContainer" containerID="19e21b1302b590c3da33c80d276973f9d9580b46f47d7d44037736c4e4dc36c8"
Jul 17 21:54:03.223375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:03.223294    2546 generic.go:356] "Generic (PLEG): container finished" podID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerID="0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412" exitCode=6
Jul 17 21:54:03.223375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:03.223346    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerDied","Data":"0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412"}
Jul 17 21:54:03.223846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:03.223417    2546 scope.go:117] "RemoveContainer" containerID="19e21b1302b590c3da33c80d276973f9d9580b46f47d7d44037736c4e4dc36c8"
Jul 17 21:54:03.223846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:03.223591    2546 scope.go:117] "RemoveContainer" containerID="0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412"
Jul 17 21:54:03.223846 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:03.223823    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd_odh-ai-gateway-infra(1f592f89-4656-425b-a1e1-da2e3ecff7a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4"
Jul 17 21:54:04.230960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:04.230927    2546 scope.go:117] "RemoveContainer" containerID="0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412"
Jul 17 21:54:04.231413 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:04.231107    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd_odh-ai-gateway-infra(1f592f89-4656-425b-a1e1-da2e3ecff7a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4"
Jul 17 21:54:04.534048 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:04.534024    2546 scope.go:117] "RemoveContainer" containerID="eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad"
Jul 17 21:54:05.243613 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:05.243579    2546 generic.go:356] "Generic (PLEG): container finished" podID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerID="9795e581c67ea8097b918ebca0987119f3cd07e5cc4bb88db92390f9e2eb6add" exitCode=7
Jul 17 21:54:05.243977 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:05.243662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerDied","Data":"9795e581c67ea8097b918ebca0987119f3cd07e5cc4bb88db92390f9e2eb6add"}
Jul 17 21:54:05.243977 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:05.243771    2546 scope.go:117] "RemoveContainer" containerID="eff9c7bae3ce0092551d716d8caed3dd073da09740b25e7ffda4d86fb45319ad"
Jul 17 21:54:05.244103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:05.244087    2546 scope.go:117] "RemoveContainer" containerID="9795e581c67ea8097b918ebca0987119f3cd07e5cc4bb88db92390f9e2eb6add"
Jul 17 21:54:05.244320 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:05.244293    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs_odh-ai-gateway-infra(11cef0ee-f8e0-4ad9-b544-950957ea523d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d"
Jul 17 21:54:06.269562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:06.269526    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:54:06.399060 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:06.399041    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:54:06.413826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:06.413616    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca\") pod \"11cef0ee-f8e0-4ad9-b544-950957ea523d\" (UID: \"11cef0ee-f8e0-4ad9-b544-950957ea523d\") "
Jul 17 21:54:06.414238 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:06.414212    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11cef0ee-f8e0-4ad9-b544-950957ea523d" (UID: "11cef0ee-f8e0-4ad9-b544-950957ea523d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:06.514612 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:06.514581    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11cef0ee-f8e0-4ad9-b544-950957ea523d-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:07.256861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:07.256829    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"
Jul 17 21:54:07.257056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:07.256836    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs" event={"ID":"11cef0ee-f8e0-4ad9-b544-950957ea523d","Type":"ContainerDied","Data":"57ae984ee6d3ec45f72ef9563e6429ea31196610bdbde99cd31c1fbee9544bb6"}
Jul 17 21:54:07.257056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:07.256964    2546 scope.go:117] "RemoveContainer" containerID="9795e581c67ea8097b918ebca0987119f3cd07e5cc4bb88db92390f9e2eb6add"
Jul 17 21:54:07.274369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:07.274345    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:54:07.281584 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:07.281558    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-hnjhs"]
Jul 17 21:54:08.479325 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479287    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:08.479810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479761    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.479810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479779    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.479810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479790    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.479810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479798    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.480030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479826    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.480030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479833    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.480030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479915    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.480030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.479929    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:08.482775 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.482754    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:08.489834 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.489641    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:08.529615 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.529588    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd\" (UID: \"adddeaea-d474-4bea-9769-9aed2f6c89f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:08.535533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.535510    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" path="/var/lib/kubelet/pods/11cef0ee-f8e0-4ad9-b544-950957ea523d/volumes"
Jul 17 21:54:08.630512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.630490    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd\" (UID: \"adddeaea-d474-4bea-9769-9aed2f6c89f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:08.631092 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.631075    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd\" (UID: \"adddeaea-d474-4bea-9769-9aed2f6c89f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:08.793820 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:08.793794    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:09.128332 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:09.128265    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:09.130599 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:09.130562    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadddeaea_d474_4bea_9769_9aed2f6c89f2.slice/crio-569bd0823376364a2a2586828829b83a7fa70182b898655eccc1be201bbba568 WatchSource:0}: Error finding container 569bd0823376364a2a2586828829b83a7fa70182b898655eccc1be201bbba568: Status 404 returned error can't find the container with id 569bd0823376364a2a2586828829b83a7fa70182b898655eccc1be201bbba568
Jul 17 21:54:09.268264 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:09.268233    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" event={"ID":"adddeaea-d474-4bea-9769-9aed2f6c89f2","Type":"ContainerStarted","Data":"569bd0823376364a2a2586828829b83a7fa70182b898655eccc1be201bbba568"}
Jul 17 21:54:10.274486 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:10.274454    2546 generic.go:356] "Generic (PLEG): container finished" podID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerID="653acbd13631e3c5d67624cfa073bef71436936ee77603e60dfbc9f8b491fc50" exitCode=6
Jul 17 21:54:10.274916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:10.274521    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" event={"ID":"adddeaea-d474-4bea-9769-9aed2f6c89f2","Type":"ContainerDied","Data":"653acbd13631e3c5d67624cfa073bef71436936ee77603e60dfbc9f8b491fc50"}
Jul 17 21:54:10.274916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:10.274692    2546 scope.go:117] "RemoveContainer" containerID="653acbd13631e3c5d67624cfa073bef71436936ee77603e60dfbc9f8b491fc50"
Jul 17 21:54:11.282001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:11.281964    2546 generic.go:356] "Generic (PLEG): container finished" podID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerID="9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751" exitCode=6
Jul 17 21:54:11.282472 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:11.282025    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" event={"ID":"adddeaea-d474-4bea-9769-9aed2f6c89f2","Type":"ContainerDied","Data":"9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751"}
Jul 17 21:54:11.282472 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:11.282131    2546 scope.go:117] "RemoveContainer" containerID="653acbd13631e3c5d67624cfa073bef71436936ee77603e60dfbc9f8b491fc50"
Jul 17 21:54:11.282472 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:11.282322    2546 scope.go:117] "RemoveContainer" containerID="9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751"
Jul 17 21:54:11.282638 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:11.282573    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd_odh-ai-gateway-infra(adddeaea-d474-4bea-9769-9aed2f6c89f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2"
Jul 17 21:54:12.291062 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:12.291032    2546 scope.go:117] "RemoveContainer" containerID="9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751"
Jul 17 21:54:12.291463 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:12.291212    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd_odh-ai-gateway-infra(adddeaea-d474-4bea-9769-9aed2f6c89f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2"
Jul 17 21:54:17.392899 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:17.392864    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:54:17.393447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:17.393077    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy" containerID="cri-o://ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84" gracePeriod=30
Jul 17 21:54:18.531111 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:18.531078    2546 scope.go:117] "RemoveContainer" containerID="0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412"
Jul 17 21:54:19.328702 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:19.328669    2546 generic.go:356] "Generic (PLEG): container finished" podID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerID="d3d1e50a2373bff044db77d0b329708ffc52a20d04dcd9a57cb715c388b8e9ee" exitCode=6
Jul 17 21:54:19.328818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:19.328746    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerDied","Data":"d3d1e50a2373bff044db77d0b329708ffc52a20d04dcd9a57cb715c388b8e9ee"}
Jul 17 21:54:19.328818 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:19.328816    2546 scope.go:117] "RemoveContainer" containerID="0c1fc704f369ad4222a8ffc3245d5ef09af172546493ef4098f3a244e7933412"
Jul 17 21:54:19.329117 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:19.329095    2546 scope.go:117] "RemoveContainer" containerID="d3d1e50a2373bff044db77d0b329708ffc52a20d04dcd9a57cb715c388b8e9ee"
Jul 17 21:54:19.329295 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:19.329273    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd_odh-ai-gateway-infra(1f592f89-4656-425b-a1e1-da2e3ecff7a4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4"
Jul 17 21:54:20.359266 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:20.359231    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:20.494483 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:20.494462    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:20.623956 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:20.623892    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca\") pod \"1f592f89-4656-425b-a1e1-da2e3ecff7a4\" (UID: \"1f592f89-4656-425b-a1e1-da2e3ecff7a4\") "
Jul 17 21:54:20.624251 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:20.624230    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1f592f89-4656-425b-a1e1-da2e3ecff7a4" (UID: "1f592f89-4656-425b-a1e1-da2e3ecff7a4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:20.724628 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:20.724595    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1f592f89-4656-425b-a1e1-da2e3ecff7a4-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:21.342006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:21.341974    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd" event={"ID":"1f592f89-4656-425b-a1e1-da2e3ecff7a4","Type":"ContainerDied","Data":"1093803789e73a79d0b85b34c09db7bdb14f798c0193100b66be4cb13ba15e32"}
Jul 17 21:54:21.342191 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:21.342012    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"
Jul 17 21:54:21.342191 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:21.342035    2546 scope.go:117] "RemoveContainer" containerID="d3d1e50a2373bff044db77d0b329708ffc52a20d04dcd9a57cb715c388b8e9ee"
Jul 17 21:54:21.364107 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:21.364084    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:21.371220 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:21.371199    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jf9kd"]
Jul 17 21:54:22.536738 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.536686    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" path="/var/lib/kubelet/pods/1f592f89-4656-425b-a1e1-da2e3ecff7a4/volumes"
Jul 17 21:54:22.659490 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.659431    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:54:22.741301 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741276    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741322    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741353    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-prk2g\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741372    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741408    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741434    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741747 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741465    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741747 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741503    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741747 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741532    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy\") pod \"6b4c5214-824a-4027-9646-aa9e951f04e8\" (UID: \"6b4c5214-824a-4027-9646-aa9e951f04e8\") "
Jul 17 21:54:22.741747 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741683    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:22.741986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741796    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.741986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.741922    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:22.742097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.742055    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:22.742097 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.742068    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:22.742215 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.742114    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data" (OuterVolumeSpecName: "istio-data") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:22.743760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.743734    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:54:22.743897 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.743793    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:54:22.744122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.744103    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g" (OuterVolumeSpecName: "kube-api-access-prk2g") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "kube-api-access-prk2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:54:22.744223 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.744143    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token" (OuterVolumeSpecName: "istio-token") pod "6b4c5214-824a-4027-9646-aa9e951f04e8" (UID: "6b4c5214-824a-4027-9646-aa9e951f04e8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:54:22.842939 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842910    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6b4c5214-824a-4027-9646-aa9e951f04e8-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.842939 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842935    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.842939 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842943    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.843327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842952    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.843327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842960    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.843327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842967    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.843327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842975    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-prk2g\" (UniqueName: \"kubernetes.io/projected/6b4c5214-824a-4027-9646-aa9e951f04e8-kube-api-access-prk2g\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:22.843327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:22.842983    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6b4c5214-824a-4027-9646-aa9e951f04e8-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:23.353876 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.353841    2546 generic.go:356] "Generic (PLEG): container finished" podID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerID="ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84" exitCode=0
Jul 17 21:54:23.354088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.353887    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"
Jul 17 21:54:23.354088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.353919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerDied","Data":"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"}
Jul 17 21:54:23.354088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.353977    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd" event={"ID":"6b4c5214-824a-4027-9646-aa9e951f04e8","Type":"ContainerDied","Data":"8206859b69ca6f137fa326d05e67545d37745eafc1dd8ac914c62f0bb85a3ead"}
Jul 17 21:54:23.354088 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.354006    2546 scope.go:117] "RemoveContainer" containerID="ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"
Jul 17 21:54:23.363197 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.363181    2546 scope.go:117] "RemoveContainer" containerID="0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"
Jul 17 21:54:23.370791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.370777    2546 scope.go:117] "RemoveContainer" containerID="ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"
Jul 17 21:54:23.371029 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:23.371011    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84\": container with ID starting with ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84 not found: ID does not exist" containerID="ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"
Jul 17 21:54:23.371072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.371038    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84"} err="failed to get container status \"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84\": rpc error: code = NotFound desc = could not find container \"ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84\": container with ID starting with ab62a112a898088eaca1be959a583883503e3e263050eedd9e8dad33e9be6d84 not found: ID does not exist"
Jul 17 21:54:23.371072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.371056    2546 scope.go:117] "RemoveContainer" containerID="0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"
Jul 17 21:54:23.371276 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:23.371260    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46\": container with ID starting with 0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46 not found: ID does not exist" containerID="0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"
Jul 17 21:54:23.371325 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.371281    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46"} err="failed to get container status \"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46\": rpc error: code = NotFound desc = could not find container \"0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46\": container with ID starting with 0299056892e5cea94430286aacfbb26822a7de3f25bc01aa24f591932d8e9a46 not found: ID does not exist"
Jul 17 21:54:23.374688 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.374667    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:54:23.380051 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:23.380028    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-b3d03bc3-openshift-default-b7f5f8975-hwxkd"]
Jul 17 21:54:24.533357 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:24.533327    2546 scope.go:117] "RemoveContainer" containerID="9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751"
Jul 17 21:54:24.536411 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:24.536386    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" path="/var/lib/kubelet/pods/6b4c5214-824a-4027-9646-aa9e951f04e8/volumes"
Jul 17 21:54:25.367277 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:25.367253    2546 generic.go:356] "Generic (PLEG): container finished" podID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerID="3f498cd80159947603bcad1d90c43806146439917c6eb22a46ae38ec7e8eb53d" exitCode=6
Jul 17 21:54:25.367405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:25.367339    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" event={"ID":"adddeaea-d474-4bea-9769-9aed2f6c89f2","Type":"ContainerDied","Data":"3f498cd80159947603bcad1d90c43806146439917c6eb22a46ae38ec7e8eb53d"}
Jul 17 21:54:25.367464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:25.367406    2546 scope.go:117] "RemoveContainer" containerID="9e23e5282f75bfccbeeb897b36aaab3d97063d7937aabb737866836f099cf751"
Jul 17 21:54:25.367714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:25.367699    2546 scope.go:117] "RemoveContainer" containerID="3f498cd80159947603bcad1d90c43806146439917c6eb22a46ae38ec7e8eb53d"
Jul 17 21:54:25.367942 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:25.367913    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd_odh-ai-gateway-infra(adddeaea-d474-4bea-9769-9aed2f6c89f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2"
Jul 17 21:54:26.394579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:26.394550    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:26.526445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:26.526425    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:26.569803 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:26.569775    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca\") pod \"adddeaea-d474-4bea-9769-9aed2f6c89f2\" (UID: \"adddeaea-d474-4bea-9769-9aed2f6c89f2\") "
Jul 17 21:54:26.570125 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:26.570102    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adddeaea-d474-4bea-9769-9aed2f6c89f2" (UID: "adddeaea-d474-4bea-9769-9aed2f6c89f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:26.671330 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:26.671261    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adddeaea-d474-4bea-9769-9aed2f6c89f2-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:27.380071 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:27.380042    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"
Jul 17 21:54:27.380253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:27.380078    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd" event={"ID":"adddeaea-d474-4bea-9769-9aed2f6c89f2","Type":"ContainerDied","Data":"569bd0823376364a2a2586828829b83a7fa70182b898655eccc1be201bbba568"}
Jul 17 21:54:27.380253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:27.380141    2546 scope.go:117] "RemoveContainer" containerID="3f498cd80159947603bcad1d90c43806146439917c6eb22a46ae38ec7e8eb53d"
Jul 17 21:54:27.406735 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:27.406695    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:27.412430 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:27.412407    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j5nvd"]
Jul 17 21:54:28.537058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:28.537010    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" path="/var/lib/kubelet/pods/adddeaea-d474-4bea-9769-9aed2f6c89f2/volumes"
Jul 17 21:54:29.517356 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517322    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517755    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517775    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517793    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517801    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517813    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517821    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517834    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517842    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517859    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517867    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517882    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.517906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517890    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517960    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517971    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517980    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="adddeaea-d474-4bea-9769-9aed2f6c89f2" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517989    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.517998    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.518005    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.518013    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.518022    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="11cef0ee-f8e0-4ad9-b544-950957ea523d" containerName="revoke-keys"
Jul 17 21:54:29.518457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.518030    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:29.522768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.522747    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.525290 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.525269    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-24f6a20e-openshift-default-dockercfg-g7f7k\""
Jul 17 21:54:29.535249 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.535231    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:54:29.592412 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592376    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592412 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592414    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kh76l\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592443    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592490    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592505    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592521    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592619    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592671    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.592843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.592701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693833 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693802    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693840    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kh76l\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693867    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693887    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693902    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693928    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.693999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.693963    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694003    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694027    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694290    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694464 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694422    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694512 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694458    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694551 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694509    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.694662 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.694641    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.696327 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.696304    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.696436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.696418    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.702130 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.702108    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kh76l\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.702433 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.702412    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token\") pod \"e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") " pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.833574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.833491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:29.969909 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.969887    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:54:29.971561 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:29.971527    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02c9f5e4_8d4c_4485_abae_dddcb77163c8.slice/crio-23866958a7ae6d577a402b96464fd4316496dad9f2c65667caae6b111bea13ce WatchSource:0}: Error finding container 23866958a7ae6d577a402b96464fd4316496dad9f2c65667caae6b111bea13ce: Status 404 returned error can't find the container with id 23866958a7ae6d577a402b96464fd4316496dad9f2c65667caae6b111bea13ce
Jul 17 21:54:29.973668 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.973641    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:29.973761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.973707    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:29.973806 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:29.973762    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:30.397043 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:30.397010    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerStarted","Data":"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"}
Jul 17 21:54:30.397207 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:30.397067    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerStarted","Data":"23866958a7ae6d577a402b96464fd4316496dad9f2c65667caae6b111bea13ce"}
Jul 17 21:54:30.423364 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:30.422985    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podStartSLOduration=1.42296552 podStartE2EDuration="1.42296552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:30.421485992 +0000 UTC m=+2246.401856468" watchObservedRunningTime="2026-07-17 21:54:30.42296552 +0000 UTC m=+2246.403336000"
Jul 17 21:54:30.833600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:30.833557    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:31.838693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:31.838661    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:32.412291 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:32.412259    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:32.413070 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:32.413051    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:35.065027 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.064972    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:35.065509 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.065423    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:35.065509 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.065444    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b4c5214-824a-4027-9646-aa9e951f04e8" containerName="istio-proxy"
Jul 17 21:54:35.065509 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.065457    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:35.065509 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.065464    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1f592f89-4656-425b-a1e1-da2e3ecff7a4" containerName="revoke-keys"
Jul 17 21:54:35.068711 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.068689    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:35.070968 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.070950    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:54:35.075530 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.075506    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:35.137846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.137814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l\" (UID: \"42a95369-518d-4609-9162-21755e45d86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:35.239191 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.239161    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l\" (UID: \"42a95369-518d-4609-9162-21755e45d86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:35.239814 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.239794    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l\" (UID: \"42a95369-518d-4609-9162-21755e45d86f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:35.381319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.381210    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:35.538713 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:35.538598    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:35.541419 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:35.541389    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42a95369_518d_4609_9162_21755e45d86f.slice/crio-bef3fd1cf3ed38e5aa8c956704eec9dfa7c321eece8da16ace39b10fa49f4060 WatchSource:0}: Error finding container bef3fd1cf3ed38e5aa8c956704eec9dfa7c321eece8da16ace39b10fa49f4060: Status 404 returned error can't find the container with id bef3fd1cf3ed38e5aa8c956704eec9dfa7c321eece8da16ace39b10fa49f4060
Jul 17 21:54:36.433665 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:36.433623    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerStarted","Data":"642ebf53e1ff583fc01364b10e3df39e8d0fd8a7997b777b415d747187d03df2"}
Jul 17 21:54:36.434190 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:36.433685    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerStarted","Data":"bef3fd1cf3ed38e5aa8c956704eec9dfa7c321eece8da16ace39b10fa49f4060"}
Jul 17 21:54:36.448473 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:36.448427    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" podStartSLOduration=1.448410835 podStartE2EDuration="1.448410835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:36.447704014 +0000 UTC m=+2252.428074492" watchObservedRunningTime="2026-07-17 21:54:36.448410835 +0000 UTC m=+2252.428781311"
Jul 17 21:54:37.440169 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.440131    2546 generic.go:356] "Generic (PLEG): container finished" podID="42a95369-518d-4609-9162-21755e45d86f" containerID="642ebf53e1ff583fc01364b10e3df39e8d0fd8a7997b777b415d747187d03df2" exitCode=7
Jul 17 21:54:37.440693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.440222    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerDied","Data":"642ebf53e1ff583fc01364b10e3df39e8d0fd8a7997b777b415d747187d03df2"}
Jul 17 21:54:37.440693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.440483    2546 scope.go:117] "RemoveContainer" containerID="642ebf53e1ff583fc01364b10e3df39e8d0fd8a7997b777b415d747187d03df2"
Jul 17 21:54:37.493166 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.493129    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:37.497650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.497629    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:37.504539 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.504496    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:37.556038 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.555999    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd\" (UID: \"91ebf5c3-c913-484f-9383-ca82b23ab90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:37.657501 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.657453    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd\" (UID: \"91ebf5c3-c913-484f-9383-ca82b23ab90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:37.658206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.658172    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd\" (UID: \"91ebf5c3-c913-484f-9383-ca82b23ab90f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:37.812190 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.812149    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:37.985783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:37.977655    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:37.990031 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:37.989998    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ebf5c3_c913_484f_9383_ca82b23ab90f.slice/crio-3b1f33990bf2c7cb64e9bfcb94e86b9f665e6ace459454e14e63ad5297502b97 WatchSource:0}: Error finding container 3b1f33990bf2c7cb64e9bfcb94e86b9f665e6ace459454e14e63ad5297502b97: Status 404 returned error can't find the container with id 3b1f33990bf2c7cb64e9bfcb94e86b9f665e6ace459454e14e63ad5297502b97
Jul 17 21:54:38.446741 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.446683    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerStarted","Data":"1b5af53ac805e8d5ca4c7d435bb009073fb3b144779b4d755bc165a94b1eb35a"}
Jul 17 21:54:38.447174 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.446804    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerStarted","Data":"3b1f33990bf2c7cb64e9bfcb94e86b9f665e6ace459454e14e63ad5297502b97"}
Jul 17 21:54:38.448642 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.448604    2546 generic.go:356] "Generic (PLEG): container finished" podID="42a95369-518d-4609-9162-21755e45d86f" containerID="d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635" exitCode=7
Jul 17 21:54:38.448785 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.448690    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerDied","Data":"d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635"}
Jul 17 21:54:38.448785 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.448763    2546 scope.go:117] "RemoveContainer" containerID="642ebf53e1ff583fc01364b10e3df39e8d0fd8a7997b777b415d747187d03df2"
Jul 17 21:54:38.448901 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.448886    2546 scope.go:117] "RemoveContainer" containerID="d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635"
Jul 17 21:54:38.449092 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:38.449073    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l_odh-ai-gateway-infra(42a95369-518d-4609-9162-21755e45d86f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" podUID="42a95369-518d-4609-9162-21755e45d86f"
Jul 17 21:54:38.465975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:38.465899    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" podStartSLOduration=1.465859638 podStartE2EDuration="1.465859638s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:38.462505628 +0000 UTC m=+2254.442876106" watchObservedRunningTime="2026-07-17 21:54:38.465859638 +0000 UTC m=+2254.446230115"
Jul 17 21:54:39.456206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:39.455655    2546 generic.go:356] "Generic (PLEG): container finished" podID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerID="1b5af53ac805e8d5ca4c7d435bb009073fb3b144779b4d755bc165a94b1eb35a" exitCode=6
Jul 17 21:54:39.456206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:39.455741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerDied","Data":"1b5af53ac805e8d5ca4c7d435bb009073fb3b144779b4d755bc165a94b1eb35a"}
Jul 17 21:54:39.456206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:39.455977    2546 scope.go:117] "RemoveContainer" containerID="1b5af53ac805e8d5ca4c7d435bb009073fb3b144779b4d755bc165a94b1eb35a"
Jul 17 21:54:39.458267 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:39.458245    2546 scope.go:117] "RemoveContainer" containerID="d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635"
Jul 17 21:54:39.458534 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:39.458500    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l_odh-ai-gateway-infra(42a95369-518d-4609-9162-21755e45d86f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" podUID="42a95369-518d-4609-9162-21755e45d86f"
Jul 17 21:54:40.466191 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:40.466098    2546 generic.go:356] "Generic (PLEG): container finished" podID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerID="b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506" exitCode=6
Jul 17 21:54:40.466672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:40.466185    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerDied","Data":"b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506"}
Jul 17 21:54:40.466672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:40.466251    2546 scope.go:117] "RemoveContainer" containerID="1b5af53ac805e8d5ca4c7d435bb009073fb3b144779b4d755bc165a94b1eb35a"
Jul 17 21:54:40.466672 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:40.466518    2546 scope.go:117] "RemoveContainer" containerID="b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506"
Jul 17 21:54:40.466875 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:40.466751    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd_odh-ai-gateway-infra(91ebf5c3-c913-484f-9383-ca82b23ab90f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f"
Jul 17 21:54:41.474838 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:41.474807    2546 scope.go:117] "RemoveContainer" containerID="b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506"
Jul 17 21:54:41.475225 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:41.475011    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd_odh-ai-gateway-infra(91ebf5c3-c913-484f-9383-ca82b23ab90f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f"
Jul 17 21:54:42.481045 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.480929    2546 generic.go:356] "Generic (PLEG): container finished" podID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerID="d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47" exitCode=0
Jul 17 21:54:42.481045 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.480992    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerDied","Data":"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"}
Jul 17 21:54:42.481621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.481461    2546 scope.go:117] "RemoveContainer" containerID="d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"
Jul 17 21:54:42.482583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.482549    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:42.482703 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.482630    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:42.482703 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:42.482664    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:54:43.497356 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:43.497320    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerStarted","Data":"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"}
Jul 17 21:54:43.834240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:43.834141    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:43.835463 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:43.835436    2546 patch_prober.go:28] interesting pod/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 17 21:54:43.835585 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:43.835486    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 17 21:54:44.834634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:44.834598    2546 patch_prober.go:28] interesting pod/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 17 21:54:44.835061 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:44.834657    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 17 21:54:45.834354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:45.834313    2546 patch_prober.go:28] interesting pod/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 17 21:54:45.834540 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:45.834373    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 17 21:54:46.833986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:46.833946    2546 patch_prober.go:28] interesting pod/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused" start-of-body=
Jul 17 21:54:46.834405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:46.834000    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.71:15021/healthz/ready\": dial tcp 10.133.0.71:15021: connect: connection refused"
Jul 17 21:54:47.412933 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:47.412890    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:47.838124 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:47.838091    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:47.839035 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:47.839015    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:54:53.531084 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:53.531051    2546 scope.go:117] "RemoveContainer" containerID="d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635"
Jul 17 21:54:54.533184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:54.533103    2546 scope.go:117] "RemoveContainer" containerID="b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506"
Jul 17 21:54:54.553935 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:54.553911    2546 generic.go:356] "Generic (PLEG): container finished" podID="42a95369-518d-4609-9162-21755e45d86f" containerID="e22a036ec3e7709b871989d662407afc65febd8ac6cd3dc16114b849950e6dee" exitCode=7
Jul 17 21:54:54.554032 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:54.553979    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerDied","Data":"e22a036ec3e7709b871989d662407afc65febd8ac6cd3dc16114b849950e6dee"}
Jul 17 21:54:54.554078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:54.554034    2546 scope.go:117] "RemoveContainer" containerID="d8c1fe5ecef91fa2495e353bb556dad0f0dfe61e78aa8140122a88b45323a635"
Jul 17 21:54:54.554250 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:54.554236    2546 scope.go:117] "RemoveContainer" containerID="e22a036ec3e7709b871989d662407afc65febd8ac6cd3dc16114b849950e6dee"
Jul 17 21:54:54.554447 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:54.554429    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l_odh-ai-gateway-infra(42a95369-518d-4609-9162-21755e45d86f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" podUID="42a95369-518d-4609-9162-21755e45d86f"
Jul 17 21:54:55.560884 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.560856    2546 generic.go:356] "Generic (PLEG): container finished" podID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerID="f7f3b329bc4325a6babb05d9e331fe00429d9274be33bac39cc3d42861557f20" exitCode=6
Jul 17 21:54:55.561316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.560921    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerDied","Data":"f7f3b329bc4325a6babb05d9e331fe00429d9274be33bac39cc3d42861557f20"}
Jul 17 21:54:55.561316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.561003    2546 scope.go:117] "RemoveContainer" containerID="b0d3dd0c5e4fc22817cdc0aa8f2b6d3a7ceb944a6e771cc61c6378a1ca6d9506"
Jul 17 21:54:55.561316 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.561298    2546 scope.go:117] "RemoveContainer" containerID="f7f3b329bc4325a6babb05d9e331fe00429d9274be33bac39cc3d42861557f20"
Jul 17 21:54:55.561537 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:54:55.561512    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd_odh-ai-gateway-infra(91ebf5c3-c913-484f-9383-ca82b23ab90f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f"
Jul 17 21:54:55.579859 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.579836    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:55.701184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.701160    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:55.827780 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.827667    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca\") pod \"42a95369-518d-4609-9162-21755e45d86f\" (UID: \"42a95369-518d-4609-9162-21755e45d86f\") "
Jul 17 21:54:55.827991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.827968    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42a95369-518d-4609-9162-21755e45d86f" (UID: "42a95369-518d-4609-9162-21755e45d86f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:55.929079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:55.929055    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42a95369-518d-4609-9162-21755e45d86f-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:56.570167 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.570139    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"
Jul 17 21:54:56.570167 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.570142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l" event={"ID":"42a95369-518d-4609-9162-21755e45d86f","Type":"ContainerDied","Data":"bef3fd1cf3ed38e5aa8c956704eec9dfa7c321eece8da16ace39b10fa49f4060"}
Jul 17 21:54:56.570583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.570203    2546 scope.go:117] "RemoveContainer" containerID="e22a036ec3e7709b871989d662407afc65febd8ac6cd3dc16114b849950e6dee"
Jul 17 21:54:56.586536 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.586517    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:56.590564 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.590544    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:56.595696 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.595671    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-dfk9l"]
Jul 17 21:54:56.717159 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.717136    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:56.836026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.835949    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca\") pod \"91ebf5c3-c913-484f-9383-ca82b23ab90f\" (UID: \"91ebf5c3-c913-484f-9383-ca82b23ab90f\") "
Jul 17 21:54:56.836281 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.836260    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "91ebf5c3-c913-484f-9383-ca82b23ab90f" (UID: "91ebf5c3-c913-484f-9383-ca82b23ab90f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:54:56.936440 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:56.936413    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91ebf5c3-c913-484f-9383-ca82b23ab90f-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:54:57.576151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:57.576124    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"
Jul 17 21:54:57.576602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:57.576150    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd" event={"ID":"91ebf5c3-c913-484f-9383-ca82b23ab90f","Type":"ContainerDied","Data":"3b1f33990bf2c7cb64e9bfcb94e86b9f665e6ace459454e14e63ad5297502b97"}
Jul 17 21:54:57.576602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:57.576211    2546 scope.go:117] "RemoveContainer" containerID="f7f3b329bc4325a6babb05d9e331fe00429d9274be33bac39cc3d42861557f20"
Jul 17 21:54:57.598775 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:57.598751    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:57.604323 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:57.604301    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gnvgd"]
Jul 17 21:54:58.513278 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513227    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:54:58.513575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513563    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513577    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513584    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513590    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513609    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513615    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513624    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513629    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513677    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513684    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513691    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513697    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.513809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.513703    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:54:58.518233 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.518211    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:54:58.520891 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.520841    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:54:58.524296 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.524272    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:54:58.537779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.537744    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42a95369-518d-4609-9162-21755e45d86f" path="/var/lib/kubelet/pods/42a95369-518d-4609-9162-21755e45d86f/volumes"
Jul 17 21:54:58.538106 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.538093    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" path="/var/lib/kubelet/pods/91ebf5c3-c913-484f-9383-ca82b23ab90f/volumes"
Jul 17 21:54:58.649817 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.649780    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr\" (UID: \"6fadcf0e-a692-481b-8970-73f5aa7c8dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:54:58.750510 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.750466    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr\" (UID: \"6fadcf0e-a692-481b-8970-73f5aa7c8dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:54:58.751128 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.751104    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr\" (UID: \"6fadcf0e-a692-481b-8970-73f5aa7c8dae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:54:58.830408 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.830319    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:54:58.969889 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:58.969866    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:54:58.972013 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:54:58.971982    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fadcf0e_a692_481b_8970_73f5aa7c8dae.slice/crio-22578648af2f5a09d8a833e99941d027c1a3691f7a3ee1665898e9adbf2630e9 WatchSource:0}: Error finding container 22578648af2f5a09d8a833e99941d027c1a3691f7a3ee1665898e9adbf2630e9: Status 404 returned error can't find the container with id 22578648af2f5a09d8a833e99941d027c1a3691f7a3ee1665898e9adbf2630e9
Jul 17 21:54:59.590256 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:59.590212    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerStarted","Data":"9b18aa58c79d94fee29f5f82b6747e710d9683c2ef88b94375d2b88a58c57805"}
Jul 17 21:54:59.590470 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:59.590303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerStarted","Data":"22578648af2f5a09d8a833e99941d027c1a3691f7a3ee1665898e9adbf2630e9"}
Jul 17 21:54:59.607135 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:54:59.607081    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" podStartSLOduration=1.60706289 podStartE2EDuration="1.60706289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:54:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:54:59.605110737 +0000 UTC m=+2275.585481213" watchObservedRunningTime="2026-07-17 21:54:59.60706289 +0000 UTC m=+2275.587433384"
Jul 17 21:55:00.341091 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341049    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:00.341507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341393    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:55:00.341507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341404    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="42a95369-518d-4609-9162-21755e45d86f" containerName="revoke-keys"
Jul 17 21:55:00.341507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341411    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:55:00.341507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341417    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:55:00.341507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.341476    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="91ebf5c3-c913-484f-9383-ca82b23ab90f" containerName="revoke-keys"
Jul 17 21:55:00.348134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.345185    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:00.353487 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.353452    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:00.464436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.464384    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d\" (UID: \"bd5448ad-2e9c-4b79-8a56-25eaededcff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:00.565430 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.565394    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d\" (UID: \"bd5448ad-2e9c-4b79-8a56-25eaededcff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:00.566048 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.566029    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d\" (UID: \"bd5448ad-2e9c-4b79-8a56-25eaededcff3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:00.596609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.596510    2546 generic.go:356] "Generic (PLEG): container finished" podID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerID="9b18aa58c79d94fee29f5f82b6747e710d9683c2ef88b94375d2b88a58c57805" exitCode=6
Jul 17 21:55:00.596609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.596582    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerDied","Data":"9b18aa58c79d94fee29f5f82b6747e710d9683c2ef88b94375d2b88a58c57805"}
Jul 17 21:55:00.596861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.596809    2546 scope.go:117] "RemoveContainer" containerID="9b18aa58c79d94fee29f5f82b6747e710d9683c2ef88b94375d2b88a58c57805"
Jul 17 21:55:00.658617 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.658576    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:00.798902 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:00.798875    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:00.800855 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:55:00.800825    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd5448ad_2e9c_4b79_8a56_25eaededcff3.slice/crio-ebc16e9fcfb4841bd46ddaa9ee3a8eb78c8e79f0d9be05e87254764febc33280 WatchSource:0}: Error finding container ebc16e9fcfb4841bd46ddaa9ee3a8eb78c8e79f0d9be05e87254764febc33280: Status 404 returned error can't find the container with id ebc16e9fcfb4841bd46ddaa9ee3a8eb78c8e79f0d9be05e87254764febc33280
Jul 17 21:55:01.603482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.603447    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerID="630b6bdde09b22627a07edfd1c950083ef457c0287196a5359d5fc26739631c4" exitCode=6
Jul 17 21:55:01.603943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.603531    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" event={"ID":"bd5448ad-2e9c-4b79-8a56-25eaededcff3","Type":"ContainerDied","Data":"630b6bdde09b22627a07edfd1c950083ef457c0287196a5359d5fc26739631c4"}
Jul 17 21:55:01.603943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.603582    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" event={"ID":"bd5448ad-2e9c-4b79-8a56-25eaededcff3","Type":"ContainerStarted","Data":"ebc16e9fcfb4841bd46ddaa9ee3a8eb78c8e79f0d9be05e87254764febc33280"}
Jul 17 21:55:01.603943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.603753    2546 scope.go:117] "RemoveContainer" containerID="630b6bdde09b22627a07edfd1c950083ef457c0287196a5359d5fc26739631c4"
Jul 17 21:55:01.605552 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.605530    2546 generic.go:356] "Generic (PLEG): container finished" podID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerID="31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed" exitCode=6
Jul 17 21:55:01.605665 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.605602    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerDied","Data":"31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed"}
Jul 17 21:55:01.605665 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.605655    2546 scope.go:117] "RemoveContainer" containerID="9b18aa58c79d94fee29f5f82b6747e710d9683c2ef88b94375d2b88a58c57805"
Jul 17 21:55:01.605842 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:01.605819    2546 scope.go:117] "RemoveContainer" containerID="31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed"
Jul 17 21:55:01.606057 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:01.606031    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr_odh-ai-gateway-infra(6fadcf0e-a692-481b-8970-73f5aa7c8dae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae"
Jul 17 21:55:02.613099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:02.613067    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerID="316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c" exitCode=6
Jul 17 21:55:02.613575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:02.613144    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" event={"ID":"bd5448ad-2e9c-4b79-8a56-25eaededcff3","Type":"ContainerDied","Data":"316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c"}
Jul 17 21:55:02.613575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:02.613197    2546 scope.go:117] "RemoveContainer" containerID="630b6bdde09b22627a07edfd1c950083ef457c0287196a5359d5fc26739631c4"
Jul 17 21:55:02.613575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:02.613384    2546 scope.go:117] "RemoveContainer" containerID="316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c"
Jul 17 21:55:02.613713 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:02.613613    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d_odh-ai-gateway-infra(bd5448ad-2e9c-4b79-8a56-25eaededcff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3"
Jul 17 21:55:02.615103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:02.615088    2546 scope.go:117] "RemoveContainer" containerID="31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed"
Jul 17 21:55:02.615241 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:02.615225    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr_odh-ai-gateway-infra(6fadcf0e-a692-481b-8970-73f5aa7c8dae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae"
Jul 17 21:55:03.621482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:03.621449    2546 scope.go:117] "RemoveContainer" containerID="316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c"
Jul 17 21:55:03.621895 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:03.621663    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d_odh-ai-gateway-infra(bd5448ad-2e9c-4b79-8a56-25eaededcff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3"
Jul 17 21:55:15.531123 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:15.531088    2546 scope.go:117] "RemoveContainer" containerID="316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c"
Jul 17 21:55:16.531003 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:16.530972    2546 scope.go:117] "RemoveContainer" containerID="31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed"
Jul 17 21:55:16.691449 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:16.691414    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerID="706c6c539344250393157486daef39c2035e10868d54519016ecf516be5824c5" exitCode=6
Jul 17 21:55:16.691904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:16.691487    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" event={"ID":"bd5448ad-2e9c-4b79-8a56-25eaededcff3","Type":"ContainerDied","Data":"706c6c539344250393157486daef39c2035e10868d54519016ecf516be5824c5"}
Jul 17 21:55:16.691904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:16.691548    2546 scope.go:117] "RemoveContainer" containerID="316a119c05f1284f80b7c7201411bd58e679df320f69f3233dd85bd604efd08c"
Jul 17 21:55:16.691904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:16.691875    2546 scope.go:117] "RemoveContainer" containerID="706c6c539344250393157486daef39c2035e10868d54519016ecf516be5824c5"
Jul 17 21:55:16.692148 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:16.692127    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d_odh-ai-gateway-infra(bd5448ad-2e9c-4b79-8a56-25eaededcff3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3"
Jul 17 21:55:17.698307 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.698277    2546 generic.go:356] "Generic (PLEG): container finished" podID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerID="4bf71ce827529b58682ae983c5f210efb9ae85dfc59b24adc8ea1cf155a30923" exitCode=6
Jul 17 21:55:17.698806 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.698358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerDied","Data":"4bf71ce827529b58682ae983c5f210efb9ae85dfc59b24adc8ea1cf155a30923"}
Jul 17 21:55:17.698806 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.698431    2546 scope.go:117] "RemoveContainer" containerID="31eed9171ca877a6614e77224b22a76bb41948a46ee9ce46ffa949b078cdc9ed"
Jul 17 21:55:17.698806 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.698744    2546 scope.go:117] "RemoveContainer" containerID="4bf71ce827529b58682ae983c5f210efb9ae85dfc59b24adc8ea1cf155a30923"
Jul 17 21:55:17.699011 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:17.698988    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr_odh-ai-gateway-infra(6fadcf0e-a692-481b-8970-73f5aa7c8dae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae"
Jul 17 21:55:17.720777 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.720752    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:17.855398 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.855369    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:17.898490 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.898458    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca\") pod \"bd5448ad-2e9c-4b79-8a56-25eaededcff3\" (UID: \"bd5448ad-2e9c-4b79-8a56-25eaededcff3\") "
Jul 17 21:55:17.898839 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.898815    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd5448ad-2e9c-4b79-8a56-25eaededcff3" (UID: "bd5448ad-2e9c-4b79-8a56-25eaededcff3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:17.999847 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:17.999817    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd5448ad-2e9c-4b79-8a56-25eaededcff3-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:55:18.706585 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.706552    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"
Jul 17 21:55:18.707050 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.706602    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d" event={"ID":"bd5448ad-2e9c-4b79-8a56-25eaededcff3","Type":"ContainerDied","Data":"ebc16e9fcfb4841bd46ddaa9ee3a8eb78c8e79f0d9be05e87254764febc33280"}
Jul 17 21:55:18.707050 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.706667    2546 scope.go:117] "RemoveContainer" containerID="706c6c539344250393157486daef39c2035e10868d54519016ecf516be5824c5"
Jul 17 21:55:18.723176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.723150    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:18.726110 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.726090    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:55:18.733134 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.733111    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kwg4d"]
Jul 17 21:55:18.857856 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.857830    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:55:18.906603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.906571    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca\") pod \"6fadcf0e-a692-481b-8970-73f5aa7c8dae\" (UID: \"6fadcf0e-a692-481b-8970-73f5aa7c8dae\") "
Jul 17 21:55:18.906975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:18.906949    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fadcf0e-a692-481b-8970-73f5aa7c8dae" (UID: "6fadcf0e-a692-481b-8970-73f5aa7c8dae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:19.008011 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.007974    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fadcf0e-a692-481b-8970-73f5aa7c8dae-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:55:19.715187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.715160    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"
Jul 17 21:55:19.715678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.715196    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr" event={"ID":"6fadcf0e-a692-481b-8970-73f5aa7c8dae","Type":"ContainerDied","Data":"22578648af2f5a09d8a833e99941d027c1a3691f7a3ee1665898e9adbf2630e9"}
Jul 17 21:55:19.715678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.715267    2546 scope.go:117] "RemoveContainer" containerID="4bf71ce827529b58682ae983c5f210efb9ae85dfc59b24adc8ea1cf155a30923"
Jul 17 21:55:19.738405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.738375    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:55:19.746283 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:19.746257    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-k6zpr"]
Jul 17 21:55:20.540991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:20.540955    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" path="/var/lib/kubelet/pods/6fadcf0e-a692-481b-8970-73f5aa7c8dae/volumes"
Jul 17 21:55:20.541284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:20.541271    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" path="/var/lib/kubelet/pods/bd5448ad-2e9c-4b79-8a56-25eaededcff3/volumes"
Jul 17 21:55:35.113225 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113193    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113503    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113514    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113523    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113528    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113538    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113544    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113556    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113560    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113606    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113615    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113621    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113627    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:35.113707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.113634    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:35.117969 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.117953    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:35.120474 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.120452    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:55:35.125421 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.125397    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:35.249715 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.249690    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f\" (UID: \"3019c247-0ee8-4df0-bd51-20dbdfc78823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:35.351001 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.350968    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f\" (UID: \"3019c247-0ee8-4df0-bd51-20dbdfc78823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:35.351559 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.351540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f\" (UID: \"3019c247-0ee8-4df0-bd51-20dbdfc78823\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:35.429253 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.429196    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:35.560362 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.560337    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:35.562992 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:55:35.562965    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3019c247_0ee8_4df0_bd51_20dbdfc78823.slice/crio-3fd011b3e2456f5a96503e4e94ccd983adbad0a51d5404757261f66e036cbb22 WatchSource:0}: Error finding container 3fd011b3e2456f5a96503e4e94ccd983adbad0a51d5404757261f66e036cbb22: Status 404 returned error can't find the container with id 3fd011b3e2456f5a96503e4e94ccd983adbad0a51d5404757261f66e036cbb22
Jul 17 21:55:35.796623 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:35.796598    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerStarted","Data":"3fd011b3e2456f5a96503e4e94ccd983adbad0a51d5404757261f66e036cbb22"}
Jul 17 21:55:36.803152 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:36.803123    2546 generic.go:356] "Generic (PLEG): container finished" podID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerID="07984760574b0af9f92f8b87572659fc2580e241d9f539cf81db87348640a63c" exitCode=7
Jul 17 21:55:36.803543 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:36.803202    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerDied","Data":"07984760574b0af9f92f8b87572659fc2580e241d9f539cf81db87348640a63c"}
Jul 17 21:55:36.803543 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:36.803359    2546 scope.go:117] "RemoveContainer" containerID="07984760574b0af9f92f8b87572659fc2580e241d9f539cf81db87348640a63c"
Jul 17 21:55:37.538337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538304    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:37.538684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538666    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:37.538813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538688    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:37.538813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538701    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:37.538813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538708    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fadcf0e-a692-481b-8970-73f5aa7c8dae" containerName="revoke-keys"
Jul 17 21:55:37.538963 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.538831    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd5448ad-2e9c-4b79-8a56-25eaededcff3" containerName="revoke-keys"
Jul 17 21:55:37.541827 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.541800    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:37.548630 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.548602    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:37.669682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.669585    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx\" (UID: \"9b81a202-a06e-4a30-86b5-cb99f9fae21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:37.770871 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.770832    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx\" (UID: \"9b81a202-a06e-4a30-86b5-cb99f9fae21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:37.771458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.771434    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx\" (UID: \"9b81a202-a06e-4a30-86b5-cb99f9fae21b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:37.809886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.809854    2546 generic.go:356] "Generic (PLEG): container finished" podID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerID="dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c" exitCode=7
Jul 17 21:55:37.810284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.809950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerDied","Data":"dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c"}
Jul 17 21:55:37.810284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.810011    2546 scope.go:117] "RemoveContainer" containerID="07984760574b0af9f92f8b87572659fc2580e241d9f539cf81db87348640a63c"
Jul 17 21:55:37.810284 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.810141    2546 scope.go:117] "RemoveContainer" containerID="dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c"
Jul 17 21:55:37.810399 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:37.810347    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f_odh-ai-gateway-infra(3019c247-0ee8-4df0-bd51-20dbdfc78823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823"
Jul 17 21:55:37.853247 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.853207    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:37.999083 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:37.999057    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:38.000907 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:55:38.000877    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b81a202_a06e_4a30_86b5_cb99f9fae21b.slice/crio-45b071de16fb2094c37a8cd77a08e5eb992e9011ec41258c7a96283f7cac9662 WatchSource:0}: Error finding container 45b071de16fb2094c37a8cd77a08e5eb992e9011ec41258c7a96283f7cac9662: Status 404 returned error can't find the container with id 45b071de16fb2094c37a8cd77a08e5eb992e9011ec41258c7a96283f7cac9662
Jul 17 21:55:38.816764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:38.816652    2546 scope.go:117] "RemoveContainer" containerID="dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c"
Jul 17 21:55:38.817205 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:38.816898    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f_odh-ai-gateway-infra(3019c247-0ee8-4df0-bd51-20dbdfc78823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823"
Jul 17 21:55:38.818203 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:38.818180    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerID="e5f3a1da23edafdf102471ccfc79f0a60b20979b22ead4020f790e60de87f756" exitCode=6
Jul 17 21:55:38.818329 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:38.818248    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerDied","Data":"e5f3a1da23edafdf102471ccfc79f0a60b20979b22ead4020f790e60de87f756"}
Jul 17 21:55:38.818329 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:38.818292    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerStarted","Data":"45b071de16fb2094c37a8cd77a08e5eb992e9011ec41258c7a96283f7cac9662"}
Jul 17 21:55:38.818439 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:38.818424    2546 scope.go:117] "RemoveContainer" containerID="e5f3a1da23edafdf102471ccfc79f0a60b20979b22ead4020f790e60de87f756"
Jul 17 21:55:39.831684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:39.831645    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerID="924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584" exitCode=6
Jul 17 21:55:39.832153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:39.831745    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerDied","Data":"924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584"}
Jul 17 21:55:39.832153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:39.831833    2546 scope.go:117] "RemoveContainer" containerID="e5f3a1da23edafdf102471ccfc79f0a60b20979b22ead4020f790e60de87f756"
Jul 17 21:55:39.832153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:39.832052    2546 scope.go:117] "RemoveContainer" containerID="924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584"
Jul 17 21:55:39.832299 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:39.832281    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx_odh-ai-gateway-infra(9b81a202-a06e-4a30-86b5-cb99f9fae21b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b"
Jul 17 21:55:40.839674 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:40.839644    2546 scope.go:117] "RemoveContainer" containerID="924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584"
Jul 17 21:55:40.840159 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:40.839940    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx_odh-ai-gateway-infra(9b81a202-a06e-4a30-86b5-cb99f9fae21b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b"
Jul 17 21:55:49.531436 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:49.531401    2546 scope.go:117] "RemoveContainer" containerID="dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c"
Jul 17 21:55:49.883710 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:49.883686    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerStarted","Data":"9d9e344391f5e8f3792f7ef311db93333e88ef48a187b8e7731a1848a98e97ad"}
Jul 17 21:55:49.900825 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:49.900778    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" podStartSLOduration=14.900762975 podStartE2EDuration="14.900762975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:49.900576355 +0000 UTC m=+2325.880946833" watchObservedRunningTime="2026-07-17 21:55:49.900762975 +0000 UTC m=+2325.881133452"
Jul 17 21:55:50.388029 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.388000    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:55:50.391194 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.391178    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:55:50.399664 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.399641    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:55:50.478463 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.478393    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv\" (UID: \"1dba961b-be16-4c15-9f3b-31a535cc8620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:55:50.579417 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.579386    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv\" (UID: \"1dba961b-be16-4c15-9f3b-31a535cc8620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:55:50.580086 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.580063    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv\" (UID: \"1dba961b-be16-4c15-9f3b-31a535cc8620\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:55:50.701663 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.701638    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:55:50.834198 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.834177    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:55:50.836228 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:55:50.836198    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dba961b_be16_4c15_9f3b_31a535cc8620.slice/crio-8fd2a615cba3ad35b4eb8a199bd64d1fbf07ef1d8f3a2ce01e5dc4c1454e4a9b WatchSource:0}: Error finding container 8fd2a615cba3ad35b4eb8a199bd64d1fbf07ef1d8f3a2ce01e5dc4c1454e4a9b: Status 404 returned error can't find the container with id 8fd2a615cba3ad35b4eb8a199bd64d1fbf07ef1d8f3a2ce01e5dc4c1454e4a9b
Jul 17 21:55:50.889686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.889658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerStarted","Data":"8fd2a615cba3ad35b4eb8a199bd64d1fbf07ef1d8f3a2ce01e5dc4c1454e4a9b"}
Jul 17 21:55:50.891396 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.891374    2546 generic.go:356] "Generic (PLEG): container finished" podID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerID="9d9e344391f5e8f3792f7ef311db93333e88ef48a187b8e7731a1848a98e97ad" exitCode=7
Jul 17 21:55:50.891500 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.891410    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerDied","Data":"9d9e344391f5e8f3792f7ef311db93333e88ef48a187b8e7731a1848a98e97ad"}
Jul 17 21:55:50.891500 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.891456    2546 scope.go:117] "RemoveContainer" containerID="dc9e7ab2ca854592753afd24eef5a9da6b1bb2b17b78122c8ddfa41bcab06f6c"
Jul 17 21:55:50.891779 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.891734    2546 scope.go:117] "RemoveContainer" containerID="9d9e344391f5e8f3792f7ef311db93333e88ef48a187b8e7731a1848a98e97ad"
Jul 17 21:55:50.891992 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:50.891969    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f_odh-ai-gateway-infra(3019c247-0ee8-4df0-bd51-20dbdfc78823)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823"
Jul 17 21:55:50.917034 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:50.917008    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:51.898044 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:51.897937    2546 generic.go:356] "Generic (PLEG): container finished" podID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerID="e86471e5cb14746afff80daea9abd6589f35e28a08d931f2fbe540c554eecce1" exitCode=6
Jul 17 21:55:51.898044 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:51.898025    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerDied","Data":"e86471e5cb14746afff80daea9abd6589f35e28a08d931f2fbe540c554eecce1"}
Jul 17 21:55:51.898597 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:51.898266    2546 scope.go:117] "RemoveContainer" containerID="e86471e5cb14746afff80daea9abd6589f35e28a08d931f2fbe540c554eecce1"
Jul 17 21:55:52.033321 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.033292    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:52.194409 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.194382    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca\") pod \"3019c247-0ee8-4df0-bd51-20dbdfc78823\" (UID: \"3019c247-0ee8-4df0-bd51-20dbdfc78823\") "
Jul 17 21:55:52.194737 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.194695    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3019c247-0ee8-4df0-bd51-20dbdfc78823" (UID: "3019c247-0ee8-4df0-bd51-20dbdfc78823"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:52.295364 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.295327    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3019c247-0ee8-4df0-bd51-20dbdfc78823-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:55:52.906749 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.906632    2546 generic.go:356] "Generic (PLEG): container finished" podID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerID="69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003" exitCode=6
Jul 17 21:55:52.906749 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.906716    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerDied","Data":"69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003"}
Jul 17 21:55:52.907293 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.906801    2546 scope.go:117] "RemoveContainer" containerID="e86471e5cb14746afff80daea9abd6589f35e28a08d931f2fbe540c554eecce1"
Jul 17 21:55:52.907293 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.906983    2546 scope.go:117] "RemoveContainer" containerID="69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003"
Jul 17 21:55:52.907293 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:52.907244    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv_odh-ai-gateway-infra(1dba961b-be16-4c15-9f3b-31a535cc8620)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620"
Jul 17 21:55:52.908205 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.908159    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"
Jul 17 21:55:52.908283 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.908212    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f" event={"ID":"3019c247-0ee8-4df0-bd51-20dbdfc78823","Type":"ContainerDied","Data":"3fd011b3e2456f5a96503e4e94ccd983adbad0a51d5404757261f66e036cbb22"}
Jul 17 21:55:52.918398 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.918380    2546 scope.go:117] "RemoveContainer" containerID="9d9e344391f5e8f3792f7ef311db93333e88ef48a187b8e7731a1848a98e97ad"
Jul 17 21:55:52.938078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.938051    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:52.944391 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:52.944369    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jzt5f"]
Jul 17 21:55:53.915248 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:53.915219    2546 scope.go:117] "RemoveContainer" containerID="69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003"
Jul 17 21:55:53.915710 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:53.915426    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv_odh-ai-gateway-infra(1dba961b-be16-4c15-9f3b-31a535cc8620)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620"
Jul 17 21:55:54.536184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:54.536149    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" path="/var/lib/kubelet/pods/3019c247-0ee8-4df0-bd51-20dbdfc78823/volumes"
Jul 17 21:55:55.531151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:55.531066    2546 scope.go:117] "RemoveContainer" containerID="924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584"
Jul 17 21:55:55.928111 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:55.928080    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerStarted","Data":"89031b6b4a3d6b642f372d981213f51a3f188547292e68ac1329be807d02d757"}
Jul 17 21:55:55.946812 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:55.946750    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" podStartSLOduration=18.946731542 podStartE2EDuration="18.946731542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:55:55.945207242 +0000 UTC m=+2331.925577742" watchObservedRunningTime="2026-07-17 21:55:55.946731542 +0000 UTC m=+2331.927102011"
Jul 17 21:55:56.941844 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:56.941809    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerID="89031b6b4a3d6b642f372d981213f51a3f188547292e68ac1329be807d02d757" exitCode=6
Jul 17 21:55:56.942260 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:56.941882    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerDied","Data":"89031b6b4a3d6b642f372d981213f51a3f188547292e68ac1329be807d02d757"}
Jul 17 21:55:56.942260 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:56.941964    2546 scope.go:117] "RemoveContainer" containerID="924cdc8067bec9f4f2361ba6d0f7981456947331db94ee8fec52ca081411e584"
Jul 17 21:55:56.942260 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:56.942232    2546 scope.go:117] "RemoveContainer" containerID="89031b6b4a3d6b642f372d981213f51a3f188547292e68ac1329be807d02d757"
Jul 17 21:55:56.942434 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:55:56.942417    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx_odh-ai-gateway-infra(9b81a202-a06e-4a30-86b5-cb99f9fae21b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b"
Jul 17 21:55:56.954850 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:56.954829    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:58.088168 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.088144    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:58.250460 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.250430    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca\") pod \"9b81a202-a06e-4a30-86b5-cb99f9fae21b\" (UID: \"9b81a202-a06e-4a30-86b5-cb99f9fae21b\") "
Jul 17 21:55:58.250785 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.250764    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b81a202-a06e-4a30-86b5-cb99f9fae21b" (UID: "9b81a202-a06e-4a30-86b5-cb99f9fae21b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:55:58.351458 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.351431    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b81a202-a06e-4a30-86b5-cb99f9fae21b-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:55:58.565902 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.565871    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:55:58.566151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566140    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566153    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566166    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566171    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566181    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566186    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566192    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566197    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566251    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566258    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.566405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566265    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:55:58.566405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.566272    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:55:58.569006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.568991    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:55:58.576539 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.576518    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:55:58.654651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.654621    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84\" (UID: \"5e67671c-e013-414e-b6f6-bd426c503f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:55:58.755535 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.755510    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84\" (UID: \"5e67671c-e013-414e-b6f6-bd426c503f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:55:58.756103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.756084    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84\" (UID: \"5e67671c-e013-414e-b6f6-bd426c503f7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:55:58.879976 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.879915    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:55:58.955700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.955669    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx" event={"ID":"9b81a202-a06e-4a30-86b5-cb99f9fae21b","Type":"ContainerDied","Data":"45b071de16fb2094c37a8cd77a08e5eb992e9011ec41258c7a96283f7cac9662"}
Jul 17 21:55:58.955700 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.955701    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"
Jul 17 21:55:58.955951 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.955763    2546 scope.go:117] "RemoveContainer" containerID="89031b6b4a3d6b642f372d981213f51a3f188547292e68ac1329be807d02d757"
Jul 17 21:55:58.974753 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.974711    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:58.978637 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:58.978614    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-z2dnx"]
Jul 17 21:55:59.020018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:59.019642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:55:59.022946 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:55:59.022917    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e67671c_e013_414e_b6f6_bd426c503f7a.slice/crio-3d291832ee81578dba1e9a355b4da4e7c84bb0e97ce05daf9077d3b51f3d4142 WatchSource:0}: Error finding container 3d291832ee81578dba1e9a355b4da4e7c84bb0e97ce05daf9077d3b51f3d4142: Status 404 returned error can't find the container with id 3d291832ee81578dba1e9a355b4da4e7c84bb0e97ce05daf9077d3b51f3d4142
Jul 17 21:55:59.964015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:59.963925    2546 generic.go:356] "Generic (PLEG): container finished" podID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerID="c82b9ae8fd07d8b2f8189f0e6f5507a8e9b1b61eb2d9a38bd67cff14fc468810" exitCode=6
Jul 17 21:55:59.964015 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:59.963980    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerDied","Data":"c82b9ae8fd07d8b2f8189f0e6f5507a8e9b1b61eb2d9a38bd67cff14fc468810"}
Jul 17 21:55:59.964447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:59.964033    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerStarted","Data":"3d291832ee81578dba1e9a355b4da4e7c84bb0e97ce05daf9077d3b51f3d4142"}
Jul 17 21:55:59.964447 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:55:59.964330    2546 scope.go:117] "RemoveContainer" containerID="c82b9ae8fd07d8b2f8189f0e6f5507a8e9b1b61eb2d9a38bd67cff14fc468810"
Jul 17 21:56:00.536019 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:00.535989    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" path="/var/lib/kubelet/pods/9b81a202-a06e-4a30-86b5-cb99f9fae21b/volumes"
Jul 17 21:56:00.970671 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:00.970584    2546 generic.go:356] "Generic (PLEG): container finished" podID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerID="83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c" exitCode=6
Jul 17 21:56:00.971054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:00.970673    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerDied","Data":"83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c"}
Jul 17 21:56:00.971054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:00.970760    2546 scope.go:117] "RemoveContainer" containerID="c82b9ae8fd07d8b2f8189f0e6f5507a8e9b1b61eb2d9a38bd67cff14fc468810"
Jul 17 21:56:00.971054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:00.970949    2546 scope.go:117] "RemoveContainer" containerID="83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c"
Jul 17 21:56:00.971180 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:00.971145    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84_odh-ai-gateway-infra(5e67671c-e013-414e-b6f6-bd426c503f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a"
Jul 17 21:56:01.977676 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:01.977642    2546 scope.go:117] "RemoveContainer" containerID="83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c"
Jul 17 21:56:01.978116 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:01.977834    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84_odh-ai-gateway-infra(5e67671c-e013-414e-b6f6-bd426c503f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a"
Jul 17 21:56:09.531274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:09.531239    2546 scope.go:117] "RemoveContainer" containerID="69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003"
Jul 17 21:56:10.018970 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:10.018940    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerStarted","Data":"06d37756dc71e965c6c63d6e132be4849d467e69d23442c94406bb5c33b6c1ed"}
Jul 17 21:56:10.039208 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:10.039126    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" podStartSLOduration=20.039112499 podStartE2EDuration="20.039112499s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:10.037342458 +0000 UTC m=+2346.017712935" watchObservedRunningTime="2026-07-17 21:56:10.039112499 +0000 UTC m=+2346.019482985"
Jul 17 21:56:11.026225 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:11.026194    2546 generic.go:356] "Generic (PLEG): container finished" podID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerID="06d37756dc71e965c6c63d6e132be4849d467e69d23442c94406bb5c33b6c1ed" exitCode=6
Jul 17 21:56:11.026651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:11.026250    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerDied","Data":"06d37756dc71e965c6c63d6e132be4849d467e69d23442c94406bb5c33b6c1ed"}
Jul 17 21:56:11.026651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:11.026298    2546 scope.go:117] "RemoveContainer" containerID="69dd57051eff6a86456eb13b842cdc5eac94f4100187cd88e1dfe9549d3ed003"
Jul 17 21:56:11.026651 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:11.026565    2546 scope.go:117] "RemoveContainer" containerID="06d37756dc71e965c6c63d6e132be4849d467e69d23442c94406bb5c33b6c1ed"
Jul 17 21:56:11.026821 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:11.026801    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv_odh-ai-gateway-infra(1dba961b-be16-4c15-9f3b-31a535cc8620)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620"
Jul 17 21:56:11.053680 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:11.053661    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:56:12.166600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:12.166579    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:56:12.263573 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:12.263543    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca\") pod \"1dba961b-be16-4c15-9f3b-31a535cc8620\" (UID: \"1dba961b-be16-4c15-9f3b-31a535cc8620\") "
Jul 17 21:56:12.263878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:12.263858    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1dba961b-be16-4c15-9f3b-31a535cc8620" (UID: "1dba961b-be16-4c15-9f3b-31a535cc8620"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:12.364505 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:12.364447    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dba961b-be16-4c15-9f3b-31a535cc8620-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:56:13.039661 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.039629    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"
Jul 17 21:56:13.039869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.039662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv" event={"ID":"1dba961b-be16-4c15-9f3b-31a535cc8620","Type":"ContainerDied","Data":"8fd2a615cba3ad35b4eb8a199bd64d1fbf07ef1d8f3a2ce01e5dc4c1454e4a9b"}
Jul 17 21:56:13.039869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.039765    2546 scope.go:117] "RemoveContainer" containerID="06d37756dc71e965c6c63d6e132be4849d467e69d23442c94406bb5c33b6c1ed"
Jul 17 21:56:13.057153 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.057100    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:56:13.059099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.059081    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-lqnjv"]
Jul 17 21:56:13.531199 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:13.531170    2546 scope.go:117] "RemoveContainer" containerID="83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c"
Jul 17 21:56:14.047246 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:14.047218    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerStarted","Data":"26408d8be98a0977b65b4196637bbf7a745438b8c93cefe03b5ba3ede4859a66"}
Jul 17 21:56:14.065582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:14.065539    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" podStartSLOduration=16.065525418 podStartE2EDuration="16.065525418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:55:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:14.062991452 +0000 UTC m=+2350.043361928" watchObservedRunningTime="2026-07-17 21:56:14.065525418 +0000 UTC m=+2350.045895895"
Jul 17 21:56:14.538041 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:14.538007    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" path="/var/lib/kubelet/pods/1dba961b-be16-4c15-9f3b-31a535cc8620/volumes"
Jul 17 21:56:15.055041 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:15.055002    2546 generic.go:356] "Generic (PLEG): container finished" podID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerID="26408d8be98a0977b65b4196637bbf7a745438b8c93cefe03b5ba3ede4859a66" exitCode=6
Jul 17 21:56:15.055041 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:15.055035    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerDied","Data":"26408d8be98a0977b65b4196637bbf7a745438b8c93cefe03b5ba3ede4859a66"}
Jul 17 21:56:15.055255 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:15.055095    2546 scope.go:117] "RemoveContainer" containerID="83d550569a9acbcc1e1884831224b49695efe4fc59d80e267fe959177035021c"
Jul 17 21:56:15.055454 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:15.055419    2546 scope.go:117] "RemoveContainer" containerID="26408d8be98a0977b65b4196637bbf7a745438b8c93cefe03b5ba3ede4859a66"
Jul 17 21:56:15.055713 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:15.055683    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84_odh-ai-gateway-infra(5e67671c-e013-414e-b6f6-bd426c503f7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a"
Jul 17 21:56:15.073889 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:15.073865    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:56:16.191212 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:16.191190    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:56:16.289455 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:16.289426    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca\") pod \"5e67671c-e013-414e-b6f6-bd426c503f7a\" (UID: \"5e67671c-e013-414e-b6f6-bd426c503f7a\") "
Jul 17 21:56:16.289833 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:16.289810    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5e67671c-e013-414e-b6f6-bd426c503f7a" (UID: "5e67671c-e013-414e-b6f6-bd426c503f7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:16.390760 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:16.390675    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e67671c-e013-414e-b6f6-bd426c503f7a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:56:17.068189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:17.068156    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"
Jul 17 21:56:17.068189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:17.068180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84" event={"ID":"5e67671c-e013-414e-b6f6-bd426c503f7a","Type":"ContainerDied","Data":"3d291832ee81578dba1e9a355b4da4e7c84bb0e97ce05daf9077d3b51f3d4142"}
Jul 17 21:56:17.068449 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:17.068249    2546 scope.go:117] "RemoveContainer" containerID="26408d8be98a0977b65b4196637bbf7a745438b8c93cefe03b5ba3ede4859a66"
Jul 17 21:56:17.096838 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:17.096811    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:56:17.102814 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:17.102795    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7bf84"]
Jul 17 21:56:18.536432 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:18.536391    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" path="/var/lib/kubelet/pods/5e67671c-e013-414e-b6f6-bd426c503f7a/volumes"
Jul 17 21:56:25.154773 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.154740    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155034    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155046    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155053    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155058    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155068    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155073    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155083    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155088    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155095    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155100    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155106    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155111    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155125    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155130    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155174    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155182    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155190    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155196    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3019c247-0ee8-4df0-bd51-20dbdfc78823" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155205    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b81a202-a06e-4a30-86b5-cb99f9fae21b" containerName="revoke-keys"
Jul 17 21:56:25.155235 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.155211    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:25.159388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.159371    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:25.161706 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.161685    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:56:25.167796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.167772    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:25.261633 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.261592    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c\" (UID: \"94ee9446-9f60-4969-b4d2-385f4f8ef77b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:25.362030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.361996    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c\" (UID: \"94ee9446-9f60-4969-b4d2-385f4f8ef77b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:25.362601 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.362582    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c\" (UID: \"94ee9446-9f60-4969-b4d2-385f4f8ef77b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:25.470365 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.470280    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:25.605694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:25.605669    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:25.607866 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:56:25.607839    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94ee9446_9f60_4969_b4d2_385f4f8ef77b.slice/crio-885e38c6d816e2be191025bbe076b1bbcf4c0f9647eedbf95d9fd69c12a74263 WatchSource:0}: Error finding container 885e38c6d816e2be191025bbe076b1bbcf4c0f9647eedbf95d9fd69c12a74263: Status 404 returned error can't find the container with id 885e38c6d816e2be191025bbe076b1bbcf4c0f9647eedbf95d9fd69c12a74263
Jul 17 21:56:26.113285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:26.113192    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerStarted","Data":"15f79bb3b6679a7c1d1ee5b856f7e98a374a8d02ebc1dbd03bf442238c6514f2"}
Jul 17 21:56:26.113285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:26.113262    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerStarted","Data":"885e38c6d816e2be191025bbe076b1bbcf4c0f9647eedbf95d9fd69c12a74263"}
Jul 17 21:56:26.129347 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:26.129297    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" podStartSLOduration=1.129280433 podStartE2EDuration="1.129280433s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:26.127834475 +0000 UTC m=+2362.108204954" watchObservedRunningTime="2026-07-17 21:56:26.129280433 +0000 UTC m=+2362.109650910"
Jul 17 21:56:27.119697 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:27.119660    2546 generic.go:356] "Generic (PLEG): container finished" podID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerID="15f79bb3b6679a7c1d1ee5b856f7e98a374a8d02ebc1dbd03bf442238c6514f2" exitCode=7
Jul 17 21:56:27.120160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:27.119748    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerDied","Data":"15f79bb3b6679a7c1d1ee5b856f7e98a374a8d02ebc1dbd03bf442238c6514f2"}
Jul 17 21:56:27.120160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:27.119953    2546 scope.go:117] "RemoveContainer" containerID="15f79bb3b6679a7c1d1ee5b856f7e98a374a8d02ebc1dbd03bf442238c6514f2"
Jul 17 21:56:28.126024 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:28.125993    2546 generic.go:356] "Generic (PLEG): container finished" podID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerID="f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82" exitCode=7
Jul 17 21:56:28.126437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:28.126072    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerDied","Data":"f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82"}
Jul 17 21:56:28.126437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:28.126127    2546 scope.go:117] "RemoveContainer" containerID="15f79bb3b6679a7c1d1ee5b856f7e98a374a8d02ebc1dbd03bf442238c6514f2"
Jul 17 21:56:28.126437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:28.126309    2546 scope.go:117] "RemoveContainer" containerID="f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82"
Jul 17 21:56:28.126570 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:28.126488    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c_odh-ai-gateway-infra(94ee9446-9f60-4969-b4d2-385f4f8ef77b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b"
Jul 17 21:56:29.132334 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:29.132303    2546 scope.go:117] "RemoveContainer" containerID="f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82"
Jul 17 21:56:29.132784 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:29.132506    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c_odh-ai-gateway-infra(94ee9446-9f60-4969-b4d2-385f4f8ef77b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b"
Jul 17 21:56:37.585394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.585353    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:37.585910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.585670    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:37.585910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.585681    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:37.585910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.585751    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e67671c-e013-414e-b6f6-bd426c503f7a" containerName="revoke-keys"
Jul 17 21:56:37.585910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.585764    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1dba961b-be16-4c15-9f3b-31a535cc8620" containerName="revoke-keys"
Jul 17 21:56:37.589858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.589833    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:37.596351 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.596260    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:37.664631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.664588    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr\" (UID: \"02766cf6-b09b-4240-8e7f-097d48051795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:37.765431 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.765396    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr\" (UID: \"02766cf6-b09b-4240-8e7f-097d48051795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:37.766050 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.766032    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr\" (UID: \"02766cf6-b09b-4240-8e7f-097d48051795\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:37.901660 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:37.901568    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:38.028446 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:38.028422    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:38.030292 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:56:38.030254    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02766cf6_b09b_4240_8e7f_097d48051795.slice/crio-85f80c456ce3a6480d3cc9d1e7347f8f365e506ace1cdea73528f00a1b8d918b WatchSource:0}: Error finding container 85f80c456ce3a6480d3cc9d1e7347f8f365e506ace1cdea73528f00a1b8d918b: Status 404 returned error can't find the container with id 85f80c456ce3a6480d3cc9d1e7347f8f365e506ace1cdea73528f00a1b8d918b
Jul 17 21:56:38.177064 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:38.177002    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" event={"ID":"02766cf6-b09b-4240-8e7f-097d48051795","Type":"ContainerStarted","Data":"85f80c456ce3a6480d3cc9d1e7347f8f365e506ace1cdea73528f00a1b8d918b"}
Jul 17 21:56:39.183670 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:39.183638    2546 generic.go:356] "Generic (PLEG): container finished" podID="02766cf6-b09b-4240-8e7f-097d48051795" containerID="b9ff55ed6ac1ec4f6a0963c6f516396056bb850eca15d006ae14d122cc7b68c7" exitCode=6
Jul 17 21:56:39.184056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:39.183752    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" event={"ID":"02766cf6-b09b-4240-8e7f-097d48051795","Type":"ContainerDied","Data":"b9ff55ed6ac1ec4f6a0963c6f516396056bb850eca15d006ae14d122cc7b68c7"}
Jul 17 21:56:39.184056 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:39.183909    2546 scope.go:117] "RemoveContainer" containerID="b9ff55ed6ac1ec4f6a0963c6f516396056bb850eca15d006ae14d122cc7b68c7"
Jul 17 21:56:40.191160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:40.191083    2546 generic.go:356] "Generic (PLEG): container finished" podID="02766cf6-b09b-4240-8e7f-097d48051795" containerID="512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13" exitCode=6
Jul 17 21:56:40.191571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:40.191166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" event={"ID":"02766cf6-b09b-4240-8e7f-097d48051795","Type":"ContainerDied","Data":"512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13"}
Jul 17 21:56:40.191571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:40.191222    2546 scope.go:117] "RemoveContainer" containerID="b9ff55ed6ac1ec4f6a0963c6f516396056bb850eca15d006ae14d122cc7b68c7"
Jul 17 21:56:40.191571 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:40.191349    2546 scope.go:117] "RemoveContainer" containerID="512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13"
Jul 17 21:56:40.191571 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:40.191550    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr_odh-ai-gateway-infra(02766cf6-b09b-4240-8e7f-097d48051795)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" podUID="02766cf6-b09b-4240-8e7f-097d48051795"
Jul 17 21:56:41.198100 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:41.198071    2546 scope.go:117] "RemoveContainer" containerID="512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13"
Jul 17 21:56:41.198476 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:41.198255    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr_odh-ai-gateway-infra(02766cf6-b09b-4240-8e7f-097d48051795)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" podUID="02766cf6-b09b-4240-8e7f-097d48051795"
Jul 17 21:56:41.531138 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:41.531115    2546 scope.go:117] "RemoveContainer" containerID="f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82"
Jul 17 21:56:42.204503 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:42.204474    2546 generic.go:356] "Generic (PLEG): container finished" podID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerID="0e2e0a0f130239297d3ad3284bf3b93578d87694fb16829ed9e41f042ce028e6" exitCode=7
Jul 17 21:56:42.204915 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:42.204530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerDied","Data":"0e2e0a0f130239297d3ad3284bf3b93578d87694fb16829ed9e41f042ce028e6"}
Jul 17 21:56:42.204915 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:42.204593    2546 scope.go:117] "RemoveContainer" containerID="f2208ec6741294c014e446e11f1c04913a4daca0f11d2261773af05256deff82"
Jul 17 21:56:42.205054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:42.204917    2546 scope.go:117] "RemoveContainer" containerID="0e2e0a0f130239297d3ad3284bf3b93578d87694fb16829ed9e41f042ce028e6"
Jul 17 21:56:42.205154 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:42.205130    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c_odh-ai-gateway-infra(94ee9446-9f60-4969-b4d2-385f4f8ef77b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b"
Jul 17 21:56:43.231257 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:43.231224    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:43.351856 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:43.351832    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:43.410748 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:43.410709    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca\") pod \"94ee9446-9f60-4969-b4d2-385f4f8ef77b\" (UID: \"94ee9446-9f60-4969-b4d2-385f4f8ef77b\") "
Jul 17 21:56:43.411051 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:43.411032    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "94ee9446-9f60-4969-b4d2-385f4f8ef77b" (UID: "94ee9446-9f60-4969-b4d2-385f4f8ef77b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:43.512103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:43.512081    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/94ee9446-9f60-4969-b4d2-385f4f8ef77b-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:56:44.217388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.217360    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"
Jul 17 21:56:44.217388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.217370    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c" event={"ID":"94ee9446-9f60-4969-b4d2-385f4f8ef77b","Type":"ContainerDied","Data":"885e38c6d816e2be191025bbe076b1bbcf4c0f9647eedbf95d9fd69c12a74263"}
Jul 17 21:56:44.217629 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.217428    2546 scope.go:117] "RemoveContainer" containerID="0e2e0a0f130239297d3ad3284bf3b93578d87694fb16829ed9e41f042ce028e6"
Jul 17 21:56:44.244591 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.244566    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:44.250006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.249985    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hvh4c"]
Jul 17 21:56:44.535602 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:44.535565    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" path="/var/lib/kubelet/pods/94ee9446-9f60-4969-b4d2-385f4f8ef77b/volumes"
Jul 17 21:56:48.616832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.616793    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617114    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617126    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617143    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617149    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617208    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617218    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.617254 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.617224    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:48.621771 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.621752    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:56:48.627399 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.627370    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:56:48.753093 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.753063    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms\" (UID: \"b6543e68-0d12-4690-b1da-a220b171aebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:56:48.854044 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.854017    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms\" (UID: \"b6543e68-0d12-4690-b1da-a220b171aebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:56:48.854582 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.854560    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms\" (UID: \"b6543e68-0d12-4690-b1da-a220b171aebe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:56:48.934333 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:48.934263    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:56:49.067503 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:49.067481    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:56:49.069629 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:56:49.069603    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6543e68_0d12_4690_b1da_a220b171aebe.slice/crio-03c91d7cc8aeba194f4fa283d2f477a6ddf7d824f307d178195484ba632eadb0 WatchSource:0}: Error finding container 03c91d7cc8aeba194f4fa283d2f477a6ddf7d824f307d178195484ba632eadb0: Status 404 returned error can't find the container with id 03c91d7cc8aeba194f4fa283d2f477a6ddf7d824f307d178195484ba632eadb0
Jul 17 21:56:49.241343 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:49.241308    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" event={"ID":"b6543e68-0d12-4690-b1da-a220b171aebe","Type":"ContainerStarted","Data":"03c91d7cc8aeba194f4fa283d2f477a6ddf7d824f307d178195484ba632eadb0"}
Jul 17 21:56:50.247714 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.247671    2546 generic.go:356] "Generic (PLEG): container finished" podID="b6543e68-0d12-4690-b1da-a220b171aebe" containerID="bf266e9512f1b34c582d04dd2402561f8f8bf404df15870059f4330b596d5495" exitCode=6
Jul 17 21:56:50.248130 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.247758    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" event={"ID":"b6543e68-0d12-4690-b1da-a220b171aebe","Type":"ContainerDied","Data":"bf266e9512f1b34c582d04dd2402561f8f8bf404df15870059f4330b596d5495"}
Jul 17 21:56:50.248130 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.248003    2546 scope.go:117] "RemoveContainer" containerID="bf266e9512f1b34c582d04dd2402561f8f8bf404df15870059f4330b596d5495"
Jul 17 21:56:50.441022 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.440993    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:56:50.441304 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.441293    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:50.441366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.441306    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="94ee9446-9f60-4969-b4d2-385f4f8ef77b" containerName="revoke-keys"
Jul 17 21:56:50.445290 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.445273    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:56:50.452562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.452395    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:56:50.567156 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.567129    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6\" (UID: \"52280138-2dd5-4430-9c7f-8a20a2692d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:56:50.667532 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.667506    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6\" (UID: \"52280138-2dd5-4430-9c7f-8a20a2692d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:56:50.668062 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.668046    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6\" (UID: \"52280138-2dd5-4430-9c7f-8a20a2692d39\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:56:50.756765 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.756743    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:56:50.886457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:50.886431    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:56:50.932490 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:56:50.932461    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52280138_2dd5_4430_9c7f_8a20a2692d39.slice/crio-64ae00695b2434597b06a4c92fc58dff2c3fc28825024f5a9997511b8c7a7190 WatchSource:0}: Error finding container 64ae00695b2434597b06a4c92fc58dff2c3fc28825024f5a9997511b8c7a7190: Status 404 returned error can't find the container with id 64ae00695b2434597b06a4c92fc58dff2c3fc28825024f5a9997511b8c7a7190
Jul 17 21:56:51.254308 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.254276    2546 generic.go:356] "Generic (PLEG): container finished" podID="b6543e68-0d12-4690-b1da-a220b171aebe" containerID="91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414" exitCode=6
Jul 17 21:56:51.254640 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.254367    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" event={"ID":"b6543e68-0d12-4690-b1da-a220b171aebe","Type":"ContainerDied","Data":"91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414"}
Jul 17 21:56:51.254640 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.254430    2546 scope.go:117] "RemoveContainer" containerID="bf266e9512f1b34c582d04dd2402561f8f8bf404df15870059f4330b596d5495"
Jul 17 21:56:51.254640 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.254605    2546 scope.go:117] "RemoveContainer" containerID="91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414"
Jul 17 21:56:51.254861 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:51.254838    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms_odh-ai-gateway-infra(b6543e68-0d12-4690-b1da-a220b171aebe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" podUID="b6543e68-0d12-4690-b1da-a220b171aebe"
Jul 17 21:56:51.256274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.256251    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerStarted","Data":"8ea2fe602a8dd0d4b93b2a368ce87e0bbb0477f5e012ab3775b40f838d9e830b"}
Jul 17 21:56:51.256361 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.256299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerStarted","Data":"64ae00695b2434597b06a4c92fc58dff2c3fc28825024f5a9997511b8c7a7190"}
Jul 17 21:56:51.288508 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.288464    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" podStartSLOduration=1.288450232 podStartE2EDuration="1.288450232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:56:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:56:51.286816981 +0000 UTC m=+2387.267187458" watchObservedRunningTime="2026-07-17 21:56:51.288450232 +0000 UTC m=+2387.268820708"
Jul 17 21:56:51.531819 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:51.531786    2546 scope.go:117] "RemoveContainer" containerID="512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13"
Jul 17 21:56:52.263104 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.263078    2546 scope.go:117] "RemoveContainer" containerID="91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414"
Jul 17 21:56:52.263475 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:52.263258    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms_odh-ai-gateway-infra(b6543e68-0d12-4690-b1da-a220b171aebe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" podUID="b6543e68-0d12-4690-b1da-a220b171aebe"
Jul 17 21:56:52.264858 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.264836    2546 generic.go:356] "Generic (PLEG): container finished" podID="02766cf6-b09b-4240-8e7f-097d48051795" containerID="ac0d5e5c4b288e0c6ccdd72cab8d2627821ae9cdeb88c70d30949f047e40140c" exitCode=6
Jul 17 21:56:52.264960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.264909    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" event={"ID":"02766cf6-b09b-4240-8e7f-097d48051795","Type":"ContainerDied","Data":"ac0d5e5c4b288e0c6ccdd72cab8d2627821ae9cdeb88c70d30949f047e40140c"}
Jul 17 21:56:52.265018 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.264973    2546 scope.go:117] "RemoveContainer" containerID="512bf2c8123b3e694687b0a128b3eabe4df3463aad7bc3d49b2737d7d0f37c13"
Jul 17 21:56:52.265270 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.265252    2546 scope.go:117] "RemoveContainer" containerID="ac0d5e5c4b288e0c6ccdd72cab8d2627821ae9cdeb88c70d30949f047e40140c"
Jul 17 21:56:52.265440 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:52.265421    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr_odh-ai-gateway-infra(02766cf6-b09b-4240-8e7f-097d48051795)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" podUID="02766cf6-b09b-4240-8e7f-097d48051795"
Jul 17 21:56:52.266533 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.266511    2546 generic.go:356] "Generic (PLEG): container finished" podID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerID="8ea2fe602a8dd0d4b93b2a368ce87e0bbb0477f5e012ab3775b40f838d9e830b" exitCode=6
Jul 17 21:56:52.266635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.266560    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerDied","Data":"8ea2fe602a8dd0d4b93b2a368ce87e0bbb0477f5e012ab3775b40f838d9e830b"}
Jul 17 21:56:52.266761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:52.266748    2546 scope.go:117] "RemoveContainer" containerID="8ea2fe602a8dd0d4b93b2a368ce87e0bbb0477f5e012ab3775b40f838d9e830b"
Jul 17 21:56:53.280245 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.280210    2546 generic.go:356] "Generic (PLEG): container finished" podID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerID="7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1" exitCode=6
Jul 17 21:56:53.280768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.280264    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerDied","Data":"7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1"}
Jul 17 21:56:53.280768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.280334    2546 scope.go:117] "RemoveContainer" containerID="8ea2fe602a8dd0d4b93b2a368ce87e0bbb0477f5e012ab3775b40f838d9e830b"
Jul 17 21:56:53.280768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.280545    2546 scope.go:117] "RemoveContainer" containerID="7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1"
Jul 17 21:56:53.280939 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:53.280772    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6_odh-ai-gateway-infra(52280138-2dd5-4430-9c7f-8a20a2692d39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39"
Jul 17 21:56:53.302166 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.302141    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:53.426992 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.426972    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:53.488751 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.488707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca\") pod \"02766cf6-b09b-4240-8e7f-097d48051795\" (UID: \"02766cf6-b09b-4240-8e7f-097d48051795\") "
Jul 17 21:56:53.489049 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.489029    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "02766cf6-b09b-4240-8e7f-097d48051795" (UID: "02766cf6-b09b-4240-8e7f-097d48051795"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:56:53.590262 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:53.590208    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/02766cf6-b09b-4240-8e7f-097d48051795-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:56:54.289092 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.289056    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"
Jul 17 21:56:54.289092 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.289078    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr" event={"ID":"02766cf6-b09b-4240-8e7f-097d48051795","Type":"ContainerDied","Data":"85f80c456ce3a6480d3cc9d1e7347f8f365e506ace1cdea73528f00a1b8d918b"}
Jul 17 21:56:54.289671 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.289155    2546 scope.go:117] "RemoveContainer" containerID="ac0d5e5c4b288e0c6ccdd72cab8d2627821ae9cdeb88c70d30949f047e40140c"
Jul 17 21:56:54.291575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.291554    2546 scope.go:117] "RemoveContainer" containerID="7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1"
Jul 17 21:56:54.291863 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:56:54.291833    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6_odh-ai-gateway-infra(52280138-2dd5-4430-9c7f-8a20a2692d39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39"
Jul 17 21:56:54.327354 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.327322    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:54.331810 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.331781    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9slqr"]
Jul 17 21:56:54.537445 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:56:54.537407    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02766cf6-b09b-4240-8e7f-097d48051795" path="/var/lib/kubelet/pods/02766cf6-b09b-4240-8e7f-097d48051795/volumes"
Jul 17 21:57:04.646678 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:04.646653    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:57:04.654634 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:04.654615    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 21:57:05.531561 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:05.531533    2546 scope.go:117] "RemoveContainer" containerID="7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1"
Jul 17 21:57:06.353338 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:06.353313    2546 generic.go:356] "Generic (PLEG): container finished" podID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerID="e16924347ff12de258aab00a5707c3eb816142a3a5bccb02042792101b1b52d6" exitCode=6
Jul 17 21:57:06.353650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:06.353394    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerDied","Data":"e16924347ff12de258aab00a5707c3eb816142a3a5bccb02042792101b1b52d6"}
Jul 17 21:57:06.353650 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:06.353458    2546 scope.go:117] "RemoveContainer" containerID="7c6921ed4dba538eba906c0c6c2fc4924400e8b718b51d9d1b92b08989e8f1c1"
Jul 17 21:57:06.353803 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:06.353713    2546 scope.go:117] "RemoveContainer" containerID="e16924347ff12de258aab00a5707c3eb816142a3a5bccb02042792101b1b52d6"
Jul 17 21:57:06.353946 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:06.353928    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6_odh-ai-gateway-infra(52280138-2dd5-4430-9c7f-8a20a2692d39)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39"
Jul 17 21:57:07.379799 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.379751    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:57:07.515246 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.515225    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:57:07.531089 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.531032    2546 scope.go:117] "RemoveContainer" containerID="91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414"
Jul 17 21:57:07.609605 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.609582    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca\") pod \"52280138-2dd5-4430-9c7f-8a20a2692d39\" (UID: \"52280138-2dd5-4430-9c7f-8a20a2692d39\") "
Jul 17 21:57:07.609923 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.609902    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "52280138-2dd5-4430-9c7f-8a20a2692d39" (UID: "52280138-2dd5-4430-9c7f-8a20a2692d39"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:07.710587 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:07.710509    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/52280138-2dd5-4430-9c7f-8a20a2692d39-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:08.366865 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.366842    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"
Jul 17 21:57:08.367021 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.366885    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6" event={"ID":"52280138-2dd5-4430-9c7f-8a20a2692d39","Type":"ContainerDied","Data":"64ae00695b2434597b06a4c92fc58dff2c3fc28825024f5a9997511b8c7a7190"}
Jul 17 21:57:08.367021 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.366942    2546 scope.go:117] "RemoveContainer" containerID="e16924347ff12de258aab00a5707c3eb816142a3a5bccb02042792101b1b52d6"
Jul 17 21:57:08.369285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.369235    2546 generic.go:356] "Generic (PLEG): container finished" podID="b6543e68-0d12-4690-b1da-a220b171aebe" containerID="c6c4aaf4497205d689375f6ed7d830ce720b88b778421ff4b6ee7427159945bb" exitCode=6
Jul 17 21:57:08.369285 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.369253    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" event={"ID":"b6543e68-0d12-4690-b1da-a220b171aebe","Type":"ContainerDied","Data":"c6c4aaf4497205d689375f6ed7d830ce720b88b778421ff4b6ee7427159945bb"}
Jul 17 21:57:08.369579 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.369563    2546 scope.go:117] "RemoveContainer" containerID="c6c4aaf4497205d689375f6ed7d830ce720b88b778421ff4b6ee7427159945bb"
Jul 17 21:57:08.369791 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:08.369771    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms_odh-ai-gateway-infra(b6543e68-0d12-4690-b1da-a220b171aebe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" podUID="b6543e68-0d12-4690-b1da-a220b171aebe"
Jul 17 21:57:08.375625 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.375589    2546 scope.go:117] "RemoveContainer" containerID="91a4aec0e7ec78da5bf39d72e6918e8774759c6c5d673568687b88f7e7eb5414"
Jul 17 21:57:08.402742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.402692    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:57:08.406478 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.406460    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-zbhk6"]
Jul 17 21:57:08.535522 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:08.535493    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" path="/var/lib/kubelet/pods/52280138-2dd5-4430-9c7f-8a20a2692d39/volumes"
Jul 17 21:57:09.399553 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:09.399520    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:57:09.528196 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:09.528174    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:57:09.622962 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:09.622941    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca\") pod \"b6543e68-0d12-4690-b1da-a220b171aebe\" (UID: \"b6543e68-0d12-4690-b1da-a220b171aebe\") "
Jul 17 21:57:09.623243 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:09.623221    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6543e68-0d12-4690-b1da-a220b171aebe" (UID: "b6543e68-0d12-4690-b1da-a220b171aebe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:09.724300 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:09.724231    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6543e68-0d12-4690-b1da-a220b171aebe-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:10.384691 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.384657    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"
Jul 17 21:57:10.384960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.384659    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms" event={"ID":"b6543e68-0d12-4690-b1da-a220b171aebe","Type":"ContainerDied","Data":"03c91d7cc8aeba194f4fa283d2f477a6ddf7d824f307d178195484ba632eadb0"}
Jul 17 21:57:10.384960 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.384801    2546 scope.go:117] "RemoveContainer" containerID="c6c4aaf4497205d689375f6ed7d830ce720b88b778421ff4b6ee7427159945bb"
Jul 17 21:57:10.410036 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.410004    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:57:10.418698 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.418675    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kqwms"]
Jul 17 21:57:10.536754 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:10.536710    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" path="/var/lib/kubelet/pods/b6543e68-0d12-4690-b1da-a220b171aebe/volumes"
Jul 17 21:57:15.221708 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.221670    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222023    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222037    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222051    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222057    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222066    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222072    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222080    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222086    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222092    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222097    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222103    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222108    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222117    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222121    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222178    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222185    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222192    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222198    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222204    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.222635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.222211    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:15.226833 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.226811    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:15.229759 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.229708    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:57:15.232079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.232043    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:15.368772 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.368710    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hm262\" (UID: \"4e15bede-7c7d-4aa0-a006-ac931e5b8e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:15.469831 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.469804    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hm262\" (UID: \"4e15bede-7c7d-4aa0-a006-ac931e5b8e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:15.470385 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.470365    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hm262\" (UID: \"4e15bede-7c7d-4aa0-a006-ac931e5b8e24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:15.539998 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.539968    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:15.670407 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:15.670378    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:15.672007 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:57:15.671979    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e15bede_7c7d_4aa0_a006_ac931e5b8e24.slice/crio-a5c7f737abd6e402838d675c18c9366c7278ad740a256943dd63bb035fb05aa8 WatchSource:0}: Error finding container a5c7f737abd6e402838d675c18c9366c7278ad740a256943dd63bb035fb05aa8: Status 404 returned error can't find the container with id a5c7f737abd6e402838d675c18c9366c7278ad740a256943dd63bb035fb05aa8
Jul 17 21:57:16.416144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:16.416109    2546 generic.go:356] "Generic (PLEG): container finished" podID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerID="8e45e105e25e28c913d49f4f38b3afb445e255aa54bebc1c242cce90b3f108fb" exitCode=7
Jul 17 21:57:16.416513 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:16.416199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" event={"ID":"4e15bede-7c7d-4aa0-a006-ac931e5b8e24","Type":"ContainerDied","Data":"8e45e105e25e28c913d49f4f38b3afb445e255aa54bebc1c242cce90b3f108fb"}
Jul 17 21:57:16.416513 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:16.416252    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" event={"ID":"4e15bede-7c7d-4aa0-a006-ac931e5b8e24","Type":"ContainerStarted","Data":"a5c7f737abd6e402838d675c18c9366c7278ad740a256943dd63bb035fb05aa8"}
Jul 17 21:57:16.416513 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:16.416322    2546 scope.go:117] "RemoveContainer" containerID="8e45e105e25e28c913d49f4f38b3afb445e255aa54bebc1c242cce90b3f108fb"
Jul 17 21:57:17.423195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:17.423163    2546 generic.go:356] "Generic (PLEG): container finished" podID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerID="45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa" exitCode=7
Jul 17 21:57:17.423574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:17.423245    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" event={"ID":"4e15bede-7c7d-4aa0-a006-ac931e5b8e24","Type":"ContainerDied","Data":"45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa"}
Jul 17 21:57:17.423574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:17.423328    2546 scope.go:117] "RemoveContainer" containerID="8e45e105e25e28c913d49f4f38b3afb445e255aa54bebc1c242cce90b3f108fb"
Jul 17 21:57:17.423574 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:17.423383    2546 scope.go:117] "RemoveContainer" containerID="45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa"
Jul 17 21:57:17.423574 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:17.423563    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hm262_odh-ai-gateway-infra(4e15bede-7c7d-4aa0-a006-ac931e5b8e24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24"
Jul 17 21:57:18.430394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:18.430361    2546 scope.go:117] "RemoveContainer" containerID="45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa"
Jul 17 21:57:18.430851 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:18.430551    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hm262_odh-ai-gateway-infra(4e15bede-7c7d-4aa0-a006-ac931e5b8e24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24"
Jul 17 21:57:27.626924 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.626883    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627233    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627244    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627254    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627260    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627317    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6543e68-0d12-4690-b1da-a220b171aebe" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627325    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="02766cf6-b09b-4240-8e7f-097d48051795" containerName="revoke-keys"
Jul 17 21:57:27.627348 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.627335    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="52280138-2dd5-4430-9c7f-8a20a2692d39" containerName="revoke-keys"
Jul 17 21:57:27.631639 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.631617    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:27.637379 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.637345    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:27.657206 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.657183    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9\" (UID: \"1256b8f8-5db8-41ce-987e-60780faed46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:27.758248 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.758220    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9\" (UID: \"1256b8f8-5db8-41ce-987e-60780faed46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:27.758816 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.758797    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9\" (UID: \"1256b8f8-5db8-41ce-987e-60780faed46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:27.945876 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:27.945816    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:28.076442 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:28.076421    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:28.078820 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:57:28.078790    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1256b8f8_5db8_41ce_987e_60780faed46e.slice/crio-ea420cbf570c4d5933cf6f73707fec9c727d67732a0a6d3065f17b45415867c2 WatchSource:0}: Error finding container ea420cbf570c4d5933cf6f73707fec9c727d67732a0a6d3065f17b45415867c2: Status 404 returned error can't find the container with id ea420cbf570c4d5933cf6f73707fec9c727d67732a0a6d3065f17b45415867c2
Jul 17 21:57:28.481969 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:28.481933    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerStarted","Data":"94ca8e2f017d08747390a2178018b286f47d1908f2f0511a3b6fadca7692fba5"}
Jul 17 21:57:28.482082 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:28.481993    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerStarted","Data":"ea420cbf570c4d5933cf6f73707fec9c727d67732a0a6d3065f17b45415867c2"}
Jul 17 21:57:28.496836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:28.496789    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" podStartSLOduration=1.496770102 podStartE2EDuration="1.496770102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:28.496465453 +0000 UTC m=+2424.476835929" watchObservedRunningTime="2026-07-17 21:57:28.496770102 +0000 UTC m=+2424.477140582"
Jul 17 21:57:29.488058 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:29.488025    2546 generic.go:356] "Generic (PLEG): container finished" podID="1256b8f8-5db8-41ce-987e-60780faed46e" containerID="94ca8e2f017d08747390a2178018b286f47d1908f2f0511a3b6fadca7692fba5" exitCode=6
Jul 17 21:57:29.488429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:29.488117    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerDied","Data":"94ca8e2f017d08747390a2178018b286f47d1908f2f0511a3b6fadca7692fba5"}
Jul 17 21:57:29.488429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:29.488249    2546 scope.go:117] "RemoveContainer" containerID="94ca8e2f017d08747390a2178018b286f47d1908f2f0511a3b6fadca7692fba5"
Jul 17 21:57:30.494872 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:30.494834    2546 generic.go:356] "Generic (PLEG): container finished" podID="1256b8f8-5db8-41ce-987e-60780faed46e" containerID="d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113" exitCode=6
Jul 17 21:57:30.495337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:30.494930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerDied","Data":"d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113"}
Jul 17 21:57:30.495337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:30.495004    2546 scope.go:117] "RemoveContainer" containerID="94ca8e2f017d08747390a2178018b286f47d1908f2f0511a3b6fadca7692fba5"
Jul 17 21:57:30.495337 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:30.495190    2546 scope.go:117] "RemoveContainer" containerID="d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113"
Jul 17 21:57:30.495473 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:30.495372    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9_odh-ai-gateway-infra(1256b8f8-5db8-41ce-987e-60780faed46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" podUID="1256b8f8-5db8-41ce-987e-60780faed46e"
Jul 17 21:57:31.501478 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:31.501451    2546 scope.go:117] "RemoveContainer" containerID="d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113"
Jul 17 21:57:31.501900 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:31.501629    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9_odh-ai-gateway-infra(1256b8f8-5db8-41ce-987e-60780faed46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" podUID="1256b8f8-5db8-41ce-987e-60780faed46e"
Jul 17 21:57:33.531781 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:33.531752    2546 scope.go:117] "RemoveContainer" containerID="45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa"
Jul 17 21:57:34.518528 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:34.518493    2546 generic.go:356] "Generic (PLEG): container finished" podID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerID="5bcdc58615db12bea8968210d980e324c3a862e41323ab4f33c65de95c0a333f" exitCode=7
Jul 17 21:57:34.518681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:34.518572    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" event={"ID":"4e15bede-7c7d-4aa0-a006-ac931e5b8e24","Type":"ContainerDied","Data":"5bcdc58615db12bea8968210d980e324c3a862e41323ab4f33c65de95c0a333f"}
Jul 17 21:57:34.518681 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:34.518667    2546 scope.go:117] "RemoveContainer" containerID="45f035eed01e8e6b00e780d9dcd15718a691024ac1a88c3e8ce00e097f57d2fa"
Jul 17 21:57:34.518944 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:34.518922    2546 scope.go:117] "RemoveContainer" containerID="5bcdc58615db12bea8968210d980e324c3a862e41323ab4f33c65de95c0a333f"
Jul 17 21:57:34.519167 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:34.519147    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hm262_odh-ai-gateway-infra(4e15bede-7c7d-4aa0-a006-ac931e5b8e24)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24"
Jul 17 21:57:35.182270 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.182243    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:57:35.182707 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.182404    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy" containerID="cri-o://4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0" gracePeriod=30
Jul 17 21:57:35.544193 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.544161    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:35.676236 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.676214    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:35.720894 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.720871    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca\") pod \"4e15bede-7c7d-4aa0-a006-ac931e5b8e24\" (UID: \"4e15bede-7c7d-4aa0-a006-ac931e5b8e24\") "
Jul 17 21:57:35.721195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.721173    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e15bede-7c7d-4aa0-a006-ac931e5b8e24" (UID: "4e15bede-7c7d-4aa0-a006-ac931e5b8e24"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:35.827785 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:35.822687    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e15bede-7c7d-4aa0-a006-ac931e5b8e24-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:36.532046 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:36.532020    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"
Jul 17 21:57:36.536157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:36.536126    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262" event={"ID":"4e15bede-7c7d-4aa0-a006-ac931e5b8e24","Type":"ContainerDied","Data":"a5c7f737abd6e402838d675c18c9366c7278ad740a256943dd63bb035fb05aa8"}
Jul 17 21:57:36.536320 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:36.536179    2546 scope.go:117] "RemoveContainer" containerID="5bcdc58615db12bea8968210d980e324c3a862e41323ab4f33c65de95c0a333f"
Jul 17 21:57:36.557201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:36.557177    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:36.561383 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:36.561361    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hm262"]
Jul 17 21:57:38.536906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:38.536869    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" path="/var/lib/kubelet/pods/4e15bede-7c7d-4aa0-a006-ac931e5b8e24/volumes"
Jul 17 21:57:40.432905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.432880    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:57:40.506081 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506054    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:57:40.506410 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506395    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.506498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506414    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.506498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506430    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.506498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506438    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.506498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506448    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:40.506498 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506456    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:40.506777 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506550    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:40.506777 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506562    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.506777 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.506572    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:40.511933 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.511914    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:57:40.520163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.519889    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:57:40.554028 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.553999    2546 generic.go:356] "Generic (PLEG): container finished" podID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerID="4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0" exitCode=0
Jul 17 21:57:40.554133 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.554043    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"
Jul 17 21:57:40.554133 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.554041    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerDied","Data":"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"}
Jul 17 21:57:40.554213 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.554158    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx" event={"ID":"02c9f5e4-8d4c-4485-abae-dddcb77163c8","Type":"ContainerDied","Data":"23866958a7ae6d577a402b96464fd4316496dad9f2c65667caae6b111bea13ce"}
Jul 17 21:57:40.554213 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.554182    2546 scope.go:117] "RemoveContainer" containerID="4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"
Jul 17 21:57:40.557874 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.557852    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.557990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.557910    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.557990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.557937    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.557990 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.557973    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558003    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558034    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558057    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558092    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kh76l\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558140    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket\") pod \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\" (UID: \"02c9f5e4-8d4c-4485-abae-dddcb77163c8\") "
Jul 17 21:57:40.558409 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558246    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:40.558409 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558320    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:40.558570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558450    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9\" (UID: \"19e01a68-242b-4b6e-9856-70ca1d8bec4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:57:40.558570 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558483    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:40.558734 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558704    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.558799 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558747    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.558799 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558761    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.559037 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.558999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:40.559037 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.559022    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:40.560578 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.560553    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 21:57:40.560701 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.560673    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:40.560851 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.560830    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l" (OuterVolumeSpecName: "kube-api-access-kh76l") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "kube-api-access-kh76l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 21:57:40.561030 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.561012    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "02c9f5e4-8d4c-4485-abae-dddcb77163c8" (UID: "02c9f5e4-8d4c-4485-abae-dddcb77163c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 21:57:40.563295 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.563270    2546 scope.go:117] "RemoveContainer" containerID="d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"
Jul 17 21:57:40.575943 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.575926    2546 scope.go:117] "RemoveContainer" containerID="4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"
Jul 17 21:57:40.576174 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:40.576157    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0\": container with ID starting with 4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0 not found: ID does not exist" containerID="4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"
Jul 17 21:57:40.576221 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.576181    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0"} err="failed to get container status \"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0\": rpc error: code = NotFound desc = could not find container \"4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0\": container with ID starting with 4bcfedf73ddb96ba3b864805128c17beea55b98fd2b082cdc7d6c765fee748f0 not found: ID does not exist"
Jul 17 21:57:40.576221 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.576197    2546 scope.go:117] "RemoveContainer" containerID="d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"
Jul 17 21:57:40.576425 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:40.576405    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47\": container with ID starting with d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47 not found: ID does not exist" containerID="d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"
Jul 17 21:57:40.576469 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.576432    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47"} err="failed to get container status \"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47\": rpc error: code = NotFound desc = could not find container \"d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47\": container with ID starting with d87199d3b7ad1abf9d30926df9bafdce05496ba00fa73ad95dabb8b3a2c8bf47 not found: ID does not exist"
Jul 17 21:57:40.659312 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659283    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9\" (UID: \"19e01a68-242b-4b6e-9856-70ca1d8bec4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659358    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659374    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659388    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659399    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659411    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/02c9f5e4-8d4c-4485-abae-dddcb77163c8-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659437 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659423    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kh76l\" (UniqueName: \"kubernetes.io/projected/02c9f5e4-8d4c-4485-abae-dddcb77163c8-kube-api-access-kh76l\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:40.659869 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.659852    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9\" (UID: \"19e01a68-242b-4b6e-9856-70ca1d8bec4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:57:40.825072 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.825010    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:57:40.902365 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.902338    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:57:40.904465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.904448    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-24f6a20e-openshift-default-68d7d8f659-vq4fx"]
Jul 17 21:57:40.959846 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:40.959815    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:57:40.962612 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:57:40.962583    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e01a68_242b_4b6e_9856_70ca1d8bec4c.slice/crio-e7fd456820b9ab916e257894e1646852ae75c7802d13ee4be289b56250157fe0 WatchSource:0}: Error finding container e7fd456820b9ab916e257894e1646852ae75c7802d13ee4be289b56250157fe0: Status 404 returned error can't find the container with id e7fd456820b9ab916e257894e1646852ae75c7802d13ee4be289b56250157fe0
Jul 17 21:57:41.560156 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:41.560119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerStarted","Data":"64912fff0d0b08d5c0b9ea5531f37c662e4c2e6d3d5efd46d363939fc2d98fc4"}
Jul 17 21:57:41.560619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:41.560175    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerStarted","Data":"e7fd456820b9ab916e257894e1646852ae75c7802d13ee4be289b56250157fe0"}
Jul 17 21:57:41.577363 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:41.577321    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" podStartSLOduration=1.577306332 podStartE2EDuration="1.577306332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:41.574847354 +0000 UTC m=+2437.555217833" watchObservedRunningTime="2026-07-17 21:57:41.577306332 +0000 UTC m=+2437.557676847"
Jul 17 21:57:42.536359 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:42.536329    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" path="/var/lib/kubelet/pods/02c9f5e4-8d4c-4485-abae-dddcb77163c8/volumes"
Jul 17 21:57:42.566491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:42.566460    2546 generic.go:356] "Generic (PLEG): container finished" podID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerID="64912fff0d0b08d5c0b9ea5531f37c662e4c2e6d3d5efd46d363939fc2d98fc4" exitCode=6
Jul 17 21:57:42.566880 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:42.566512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerDied","Data":"64912fff0d0b08d5c0b9ea5531f37c662e4c2e6d3d5efd46d363939fc2d98fc4"}
Jul 17 21:57:42.566880 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:42.566746    2546 scope.go:117] "RemoveContainer" containerID="64912fff0d0b08d5c0b9ea5531f37c662e4c2e6d3d5efd46d363939fc2d98fc4"
Jul 17 21:57:43.573405 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:43.573371    2546 generic.go:356] "Generic (PLEG): container finished" podID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerID="adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392" exitCode=6
Jul 17 21:57:43.573791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:43.573458    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerDied","Data":"adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392"}
Jul 17 21:57:43.573791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:43.573526    2546 scope.go:117] "RemoveContainer" containerID="64912fff0d0b08d5c0b9ea5531f37c662e4c2e6d3d5efd46d363939fc2d98fc4"
Jul 17 21:57:43.573791 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:43.573752    2546 scope.go:117] "RemoveContainer" containerID="adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392"
Jul 17 21:57:43.573991 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:43.573972    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9_odh-ai-gateway-infra(19e01a68-242b-4b6e-9856-70ca1d8bec4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c"
Jul 17 21:57:44.579394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:44.579368    2546 scope.go:117] "RemoveContainer" containerID="adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392"
Jul 17 21:57:44.579832 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:44.579585    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9_odh-ai-gateway-infra(19e01a68-242b-4b6e-9856-70ca1d8bec4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c"
Jul 17 21:57:46.537257 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:46.537220    2546 scope.go:117] "RemoveContainer" containerID="d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113"
Jul 17 21:57:47.595864 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:47.595830    2546 generic.go:356] "Generic (PLEG): container finished" podID="1256b8f8-5db8-41ce-987e-60780faed46e" containerID="f9d65d2c8eba1aac0b06b4cf02c430a98b38b1597689a26b8d83d80a4bc94717" exitCode=6
Jul 17 21:57:47.596261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:47.595906    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerDied","Data":"f9d65d2c8eba1aac0b06b4cf02c430a98b38b1597689a26b8d83d80a4bc94717"}
Jul 17 21:57:47.596261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:47.595980    2546 scope.go:117] "RemoveContainer" containerID="d6ce408b548c8cf00978958d015c7147e2735aad0cd3d2873695f57ac3407113"
Jul 17 21:57:47.596386 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:47.596298    2546 scope.go:117] "RemoveContainer" containerID="f9d65d2c8eba1aac0b06b4cf02c430a98b38b1597689a26b8d83d80a4bc94717"
Jul 17 21:57:47.596518 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:47.596498    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9_odh-ai-gateway-infra(1256b8f8-5db8-41ce-987e-60780faed46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" podUID="1256b8f8-5db8-41ce-987e-60780faed46e"
Jul 17 21:57:48.636438 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.636405    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:48.677491 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677453    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:57:48.677813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677794    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:48.677813 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677813    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:48.677916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677822    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:48.677916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677828    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:48.677916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677902    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="02c9f5e4-8d4c-4485-abae-dddcb77163c8" containerName="istio-proxy"
Jul 17 21:57:48.677916 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.677912    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e15bede-7c7d-4aa0-a006-ac931e5b8e24" containerName="revoke-keys"
Jul 17 21:57:48.682078 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.682057    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:57:48.690904 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.690884    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:57:48.790150 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.790126    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:48.820682 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.820657    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq\" (UID: \"acf6d9ce-059b-46d9-bae0-bddcb1a510db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:57:48.921983 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.921899    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca\") pod \"1256b8f8-5db8-41ce-987e-60780faed46e\" (UID: \"1256b8f8-5db8-41ce-987e-60780faed46e\") "
Jul 17 21:57:48.922137 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.922006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq\" (UID: \"acf6d9ce-059b-46d9-bae0-bddcb1a510db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:57:48.922323 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.922298    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1256b8f8-5db8-41ce-987e-60780faed46e" (UID: "1256b8f8-5db8-41ce-987e-60780faed46e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:57:48.922608 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.922590    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq\" (UID: \"acf6d9ce-059b-46d9-bae0-bddcb1a510db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:57:48.994737 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:48.994701    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:57:49.022895 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.022867    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1256b8f8-5db8-41ce-987e-60780faed46e-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:57:49.331545 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.331519    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:57:49.333902 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:57:49.333866    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf6d9ce_059b_46d9_bae0_bddcb1a510db.slice/crio-be378bf4e901218957825629d4602fb791a567dcf271d74c8ac02864b448cab1 WatchSource:0}: Error finding container be378bf4e901218957825629d4602fb791a567dcf271d74c8ac02864b448cab1: Status 404 returned error can't find the container with id be378bf4e901218957825629d4602fb791a567dcf271d74c8ac02864b448cab1
Jul 17 21:57:49.609195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.609119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerStarted","Data":"8d9932e057e4b6d693bbe7ea0c68d8f42d114bcaa5ce6b5056b61a5caae3cd90"}
Jul 17 21:57:49.609195 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.609179    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerStarted","Data":"be378bf4e901218957825629d4602fb791a567dcf271d74c8ac02864b448cab1"}
Jul 17 21:57:49.610406 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.610386    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"
Jul 17 21:57:49.610496 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.610377    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9" event={"ID":"1256b8f8-5db8-41ce-987e-60780faed46e","Type":"ContainerDied","Data":"ea420cbf570c4d5933cf6f73707fec9c727d67732a0a6d3065f17b45415867c2"}
Jul 17 21:57:49.610555 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.610507    2546 scope.go:117] "RemoveContainer" containerID="f9d65d2c8eba1aac0b06b4cf02c430a98b38b1597689a26b8d83d80a4bc94717"
Jul 17 21:57:49.624267 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.624233    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" podStartSLOduration=1.624220519 podStartE2EDuration="1.624220519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:57:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:57:49.623229552 +0000 UTC m=+2445.603600031" watchObservedRunningTime="2026-07-17 21:57:49.624220519 +0000 UTC m=+2445.604590996"
Jul 17 21:57:49.636244 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.636220    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:49.645843 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:49.645816    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-9x8j9"]
Jul 17 21:57:50.536581 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:50.536545    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" path="/var/lib/kubelet/pods/1256b8f8-5db8-41ce-987e-60780faed46e/volumes"
Jul 17 21:57:50.617709 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:50.617682    2546 generic.go:356] "Generic (PLEG): container finished" podID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerID="8d9932e057e4b6d693bbe7ea0c68d8f42d114bcaa5ce6b5056b61a5caae3cd90" exitCode=6
Jul 17 21:57:50.617877 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:50.617755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerDied","Data":"8d9932e057e4b6d693bbe7ea0c68d8f42d114bcaa5ce6b5056b61a5caae3cd90"}
Jul 17 21:57:50.617955 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:50.617942    2546 scope.go:117] "RemoveContainer" containerID="8d9932e057e4b6d693bbe7ea0c68d8f42d114bcaa5ce6b5056b61a5caae3cd90"
Jul 17 21:57:51.625635 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:51.625597    2546 generic.go:356] "Generic (PLEG): container finished" podID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerID="d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647" exitCode=6
Jul 17 21:57:51.626101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:51.625681    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerDied","Data":"d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647"}
Jul 17 21:57:51.626101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:51.625836    2546 scope.go:117] "RemoveContainer" containerID="8d9932e057e4b6d693bbe7ea0c68d8f42d114bcaa5ce6b5056b61a5caae3cd90"
Jul 17 21:57:51.626101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:51.625979    2546 scope.go:117] "RemoveContainer" containerID="d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647"
Jul 17 21:57:51.626231 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:51.626182    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq_odh-ai-gateway-infra(acf6d9ce-059b-46d9-bae0-bddcb1a510db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db"
Jul 17 21:57:52.632371 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:52.632348    2546 scope.go:117] "RemoveContainer" containerID="d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647"
Jul 17 21:57:52.632823 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:57:52.632510    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq_odh-ai-gateway-infra(acf6d9ce-059b-46d9-bae0-bddcb1a510db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db"
Jul 17 21:57:59.530764 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:57:59.530712    2546 scope.go:117] "RemoveContainer" containerID="adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392"
Jul 17 21:58:00.674532 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:00.674499    2546 generic.go:356] "Generic (PLEG): container finished" podID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerID="45f72d7b293513aa1d2b140d330cb9bfabc3405aaccc63048942f1f8a8ec5d8f" exitCode=6
Jul 17 21:58:00.674978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:00.674572    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerDied","Data":"45f72d7b293513aa1d2b140d330cb9bfabc3405aaccc63048942f1f8a8ec5d8f"}
Jul 17 21:58:00.674978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:00.674636    2546 scope.go:117] "RemoveContainer" containerID="adbf4e639c7f9d61e798ba9b68766f9b6343e3e936a545e6bcfc80e3a84b9392"
Jul 17 21:58:00.674978 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:00.674924    2546 scope.go:117] "RemoveContainer" containerID="45f72d7b293513aa1d2b140d330cb9bfabc3405aaccc63048942f1f8a8ec5d8f"
Jul 17 21:58:00.675176 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:00.675148    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9_odh-ai-gateway-infra(19e01a68-242b-4b6e-9856-70ca1d8bec4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c"
Jul 17 21:58:01.700264 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:01.700232    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:58:01.833561 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:01.833536    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:58:01.924684 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:01.924647    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca\") pod \"19e01a68-242b-4b6e-9856-70ca1d8bec4c\" (UID: \"19e01a68-242b-4b6e-9856-70ca1d8bec4c\") "
Jul 17 21:58:01.925023 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:01.924999    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "19e01a68-242b-4b6e-9856-70ca1d8bec4c" (UID: "19e01a68-242b-4b6e-9856-70ca1d8bec4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:02.026073 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.026028    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/19e01a68-242b-4b6e-9856-70ca1d8bec4c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:58:02.687648 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.687617    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"
Jul 17 21:58:02.687648 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.687630    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9" event={"ID":"19e01a68-242b-4b6e-9856-70ca1d8bec4c","Type":"ContainerDied","Data":"e7fd456820b9ab916e257894e1646852ae75c7802d13ee4be289b56250157fe0"}
Jul 17 21:58:02.687914 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.687705    2546 scope.go:117] "RemoveContainer" containerID="45f72d7b293513aa1d2b140d330cb9bfabc3405aaccc63048942f1f8a8ec5d8f"
Jul 17 21:58:02.704037 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.704014    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:58:02.710482 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:02.710459    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-gszt9"]
Jul 17 21:58:04.536802 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:04.536766    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" path="/var/lib/kubelet/pods/19e01a68-242b-4b6e-9856-70ca1d8bec4c/volumes"
Jul 17 21:58:04.537514 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:04.537498    2546 scope.go:117] "RemoveContainer" containerID="d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647"
Jul 17 21:58:05.704489 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:05.704464    2546 generic.go:356] "Generic (PLEG): container finished" podID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerID="5a637a4f6df25497bf8b956e0b9bd472b1fb63971a03699b4b0c45d3ed9d1a93" exitCode=6
Jul 17 21:58:05.704834 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:05.704542    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerDied","Data":"5a637a4f6df25497bf8b956e0b9bd472b1fb63971a03699b4b0c45d3ed9d1a93"}
Jul 17 21:58:05.704834 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:05.704611    2546 scope.go:117] "RemoveContainer" containerID="d8db57d60777d8283558ea360ce81d6319709df95861fc5202a7c9fdb997e647"
Jul 17 21:58:05.704995 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:05.704903    2546 scope.go:117] "RemoveContainer" containerID="5a637a4f6df25497bf8b956e0b9bd472b1fb63971a03699b4b0c45d3ed9d1a93"
Jul 17 21:58:05.705080 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:05.705059    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq_odh-ai-gateway-infra(acf6d9ce-059b-46d9-bae0-bddcb1a510db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db"
Jul 17 21:58:06.731168 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:06.731135    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:58:06.874790 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:06.874769    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:58:06.967224 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:06.967202    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca\") pod \"acf6d9ce-059b-46d9-bae0-bddcb1a510db\" (UID: \"acf6d9ce-059b-46d9-bae0-bddcb1a510db\") "
Jul 17 21:58:06.967513 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:06.967493    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "acf6d9ce-059b-46d9-bae0-bddcb1a510db" (UID: "acf6d9ce-059b-46d9-bae0-bddcb1a510db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:07.068274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.068251    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/acf6d9ce-059b-46d9-bae0-bddcb1a510db-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:58:07.716240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.716206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq" event={"ID":"acf6d9ce-059b-46d9-bae0-bddcb1a510db","Type":"ContainerDied","Data":"be378bf4e901218957825629d4602fb791a567dcf271d74c8ac02864b448cab1"}
Jul 17 21:58:07.716240 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.716240    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"
Jul 17 21:58:07.716451 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.716273    2546 scope.go:117] "RemoveContainer" containerID="5a637a4f6df25497bf8b956e0b9bd472b1fb63971a03699b4b0c45d3ed9d1a93"
Jul 17 21:58:07.737368 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.737346    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:58:07.744836 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:07.744808    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jsvkq"]
Jul 17 21:58:08.535210 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:08.535181    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" path="/var/lib/kubelet/pods/acf6d9ce-059b-46d9-bae0-bddcb1a510db/volumes"
Jul 17 21:58:15.269144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269112    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269398    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269410    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269423    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269432    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269442    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269447    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269453    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269458    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269469    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269474    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269479    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269484    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269496    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269504    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269553    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269561    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269566    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269572    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269576    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269583    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269592    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.269600 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.269599    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1256b8f8-5db8-41ce-987e-60780faed46e" containerName="revoke-keys"
Jul 17 21:58:15.272418 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.272403    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:15.274742 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.274697    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:58:15.280261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.279785    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:15.424204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.424179    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd\" (UID: \"316d79a9-9105-46b0-8b12-92ae29ef2deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:15.525187 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.525125    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd\" (UID: \"316d79a9-9105-46b0-8b12-92ae29ef2deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:15.525624 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.525608    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd\" (UID: \"316d79a9-9105-46b0-8b12-92ae29ef2deb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:15.586369 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.586352    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:15.713185 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.713153    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:15.714816 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:58:15.714791    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod316d79a9_9105_46b0_8b12_92ae29ef2deb.slice/crio-992f260b3ae52be83271f9d95efedd6f1fb45074822de59d28f2b7bd71926df4 WatchSource:0}: Error finding container 992f260b3ae52be83271f9d95efedd6f1fb45074822de59d28f2b7bd71926df4: Status 404 returned error can't find the container with id 992f260b3ae52be83271f9d95efedd6f1fb45074822de59d28f2b7bd71926df4
Jul 17 21:58:15.754238 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:15.754210    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerStarted","Data":"992f260b3ae52be83271f9d95efedd6f1fb45074822de59d28f2b7bd71926df4"}
Jul 17 21:58:16.760563 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:16.760530    2546 generic.go:356] "Generic (PLEG): container finished" podID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerID="4448794726c36dcae4e6884c9f8082e14bf8f25bb717e96f3295dfbb8bf96a20" exitCode=6
Jul 17 21:58:16.760930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:16.760570    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerDied","Data":"4448794726c36dcae4e6884c9f8082e14bf8f25bb717e96f3295dfbb8bf96a20"}
Jul 17 21:58:16.760930 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:16.760754    2546 scope.go:117] "RemoveContainer" containerID="4448794726c36dcae4e6884c9f8082e14bf8f25bb717e96f3295dfbb8bf96a20"
Jul 17 21:58:17.767090 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:17.767064    2546 generic.go:356] "Generic (PLEG): container finished" podID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerID="ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792" exitCode=6
Jul 17 21:58:17.767467 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:17.767196    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerDied","Data":"ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792"}
Jul 17 21:58:17.767467 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:17.767252    2546 scope.go:117] "RemoveContainer" containerID="4448794726c36dcae4e6884c9f8082e14bf8f25bb717e96f3295dfbb8bf96a20"
Jul 17 21:58:17.767467 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:17.767269    2546 scope.go:117] "RemoveContainer" containerID="ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792"
Jul 17 21:58:17.767467 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:17.767411    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd_odh-ai-gateway-infra(316d79a9-9105-46b0-8b12-92ae29ef2deb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb"
Jul 17 21:58:18.772988 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:18.772956    2546 scope.go:117] "RemoveContainer" containerID="ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792"
Jul 17 21:58:18.773356 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:18.773141    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd_odh-ai-gateway-infra(316d79a9-9105-46b0-8b12-92ae29ef2deb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb"
Jul 17 21:58:27.715155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715118    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:27.715534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715401    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:27.715534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715410    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="acf6d9ce-059b-46d9-bae0-bddcb1a510db" containerName="revoke-keys"
Jul 17 21:58:27.715534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715430    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:27.715534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715435    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:27.715534 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.715486    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="19e01a68-242b-4b6e-9856-70ca1d8bec4c" containerName="revoke-keys"
Jul 17 21:58:27.719619 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.719603    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:27.730462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.730435    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:27.818378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.814148    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf\" (UID: \"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:27.915375 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.915352    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf\" (UID: \"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:27.915906 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:27.915891    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf\" (UID: \"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:28.030098 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.030072    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:28.154981 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.154952    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:28.156389 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:58:28.156360    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea2d6c6b_7e31_4f8a_8cbb_4fbc7fde87d9.slice/crio-aa3020150ee4cacf662f70d7a1b2b0eca25494c7f39337143e494e28f24b71fe WatchSource:0}: Error finding container aa3020150ee4cacf662f70d7a1b2b0eca25494c7f39337143e494e28f24b71fe: Status 404 returned error can't find the container with id aa3020150ee4cacf662f70d7a1b2b0eca25494c7f39337143e494e28f24b71fe
Jul 17 21:58:28.822435 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.822404    2546 generic.go:356] "Generic (PLEG): container finished" podID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerID="1f45bcebdc739955afeb6a88e6bc6af9dfbe47d95a85b453d921dff7576d7fbb" exitCode=6
Jul 17 21:58:28.822829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.822490    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerDied","Data":"1f45bcebdc739955afeb6a88e6bc6af9dfbe47d95a85b453d921dff7576d7fbb"}
Jul 17 21:58:28.822829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.822537    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerStarted","Data":"aa3020150ee4cacf662f70d7a1b2b0eca25494c7f39337143e494e28f24b71fe"}
Jul 17 21:58:28.822829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:28.822616    2546 scope.go:117] "RemoveContainer" containerID="1f45bcebdc739955afeb6a88e6bc6af9dfbe47d95a85b453d921dff7576d7fbb"
Jul 17 21:58:29.828507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:29.828471    2546 generic.go:356] "Generic (PLEG): container finished" podID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerID="bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd" exitCode=6
Jul 17 21:58:29.828938 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:29.828515    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerDied","Data":"bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd"}
Jul 17 21:58:29.828938 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:29.828573    2546 scope.go:117] "RemoveContainer" containerID="1f45bcebdc739955afeb6a88e6bc6af9dfbe47d95a85b453d921dff7576d7fbb"
Jul 17 21:58:29.828938 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:29.828775    2546 scope.go:117] "RemoveContainer" containerID="bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd"
Jul 17 21:58:29.829056 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:29.828977    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf_odh-ai-gateway-infra(ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9"
Jul 17 21:58:30.834627 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:30.834604    2546 scope.go:117] "RemoveContainer" containerID="bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd"
Jul 17 21:58:30.834998 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:30.834768    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf_odh-ai-gateway-infra(ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9"
Jul 17 21:58:34.533507 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:34.533477    2546 scope.go:117] "RemoveContainer" containerID="ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792"
Jul 17 21:58:34.859669 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:34.859639    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerStarted","Data":"8a057f07a4dfd93ab8282bdb417dbf4371c6f762528bd6e279eaee388216d9ce"}
Jul 17 21:58:34.876261 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:34.876217    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" podStartSLOduration=19.876203429 podStartE2EDuration="19.876203429s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:34.87512727 +0000 UTC m=+2490.855497745" watchObservedRunningTime="2026-07-17 21:58:34.876203429 +0000 UTC m=+2490.856573904"
Jul 17 21:58:35.866463 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:35.866426    2546 generic.go:356] "Generic (PLEG): container finished" podID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerID="8a057f07a4dfd93ab8282bdb417dbf4371c6f762528bd6e279eaee388216d9ce" exitCode=6
Jul 17 21:58:35.866912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:35.866494    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerDied","Data":"8a057f07a4dfd93ab8282bdb417dbf4371c6f762528bd6e279eaee388216d9ce"}
Jul 17 21:58:35.866912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:35.866545    2546 scope.go:117] "RemoveContainer" containerID="ac1fe2e652c6db5fdac6def61a0fe2724d25c76b764a6aa05766a6e431e4e792"
Jul 17 21:58:35.866912 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:35.866849    2546 scope.go:117] "RemoveContainer" containerID="8a057f07a4dfd93ab8282bdb417dbf4371c6f762528bd6e279eaee388216d9ce"
Jul 17 21:58:35.867081 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:35.867063    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd_odh-ai-gateway-infra(316d79a9-9105-46b0-8b12-92ae29ef2deb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb"
Jul 17 21:58:35.890879 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:35.890856    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:36.998503 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:36.998481    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:37.077687 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.077648    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca\") pod \"316d79a9-9105-46b0-8b12-92ae29ef2deb\" (UID: \"316d79a9-9105-46b0-8b12-92ae29ef2deb\") "
Jul 17 21:58:37.078032 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.078009    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "316d79a9-9105-46b0-8b12-92ae29ef2deb" (UID: "316d79a9-9105-46b0-8b12-92ae29ef2deb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:37.178575 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.178513    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/316d79a9-9105-46b0-8b12-92ae29ef2deb-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:58:37.877920 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.877891    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"
Jul 17 21:58:37.878109 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.877935    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd" event={"ID":"316d79a9-9105-46b0-8b12-92ae29ef2deb","Type":"ContainerDied","Data":"992f260b3ae52be83271f9d95efedd6f1fb45074822de59d28f2b7bd71926df4"}
Jul 17 21:58:37.878109 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.877998    2546 scope.go:117] "RemoveContainer" containerID="8a057f07a4dfd93ab8282bdb417dbf4371c6f762528bd6e279eaee388216d9ce"
Jul 17 21:58:37.898422 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.898402    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:37.904562 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:37.904533    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-qrcwd"]
Jul 17 21:58:38.535609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.535572    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" path="/var/lib/kubelet/pods/316d79a9-9105-46b0-8b12-92ae29ef2deb/volumes"
Jul 17 21:58:38.714409 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714383    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:38.714666 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714655    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.714736 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714668    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.714736 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714686    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.714736 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714691    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.714826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714749    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.714826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.714757    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:38.718868 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.718852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:38.726053 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.726021    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:38.789992 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.789925    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst\" (UID: \"bd344f19-d660-4878-ac04-3ac9b158bc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:38.891157 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.891131    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst\" (UID: \"bd344f19-d660-4878-ac04-3ac9b158bc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:38.891671 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:38.891655    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst\" (UID: \"bd344f19-d660-4878-ac04-3ac9b158bc8a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:39.029761 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.029735    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:39.158548 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.158527    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:39.160196 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:58:39.160169    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd344f19_d660_4878_ac04_3ac9b158bc8a.slice/crio-2c5d5858cc71a4018bd69879a31bb974e19173fe058c4f2ff0b9cbc1e67dd63d WatchSource:0}: Error finding container 2c5d5858cc71a4018bd69879a31bb974e19173fe058c4f2ff0b9cbc1e67dd63d: Status 404 returned error can't find the container with id 2c5d5858cc71a4018bd69879a31bb974e19173fe058c4f2ff0b9cbc1e67dd63d
Jul 17 21:58:39.888892 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.888859    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerID="f46a8551340fffd4f96a131125b53e7d3e744f647a5c3e972b9cc403eabef839" exitCode=6
Jul 17 21:58:39.889201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.888939    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerDied","Data":"f46a8551340fffd4f96a131125b53e7d3e744f647a5c3e972b9cc403eabef839"}
Jul 17 21:58:39.889201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.888981    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerStarted","Data":"2c5d5858cc71a4018bd69879a31bb974e19173fe058c4f2ff0b9cbc1e67dd63d"}
Jul 17 21:58:39.889201 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:39.889072    2546 scope.go:117] "RemoveContainer" containerID="f46a8551340fffd4f96a131125b53e7d3e744f647a5c3e972b9cc403eabef839"
Jul 17 21:58:40.552862 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.552833    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:58:40.553125 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.553114    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:40.553170 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.553126    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:40.553204 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.553180    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="316d79a9-9105-46b0-8b12-92ae29ef2deb" containerName="revoke-keys"
Jul 17 21:58:40.555888 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.555870    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:58:40.564151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.564124    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:58:40.601431 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.601402    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj\" (UID: \"e78ba784-3877-4627-bbd5-b65bdb734faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:58:40.702768 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.702733    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj\" (UID: \"e78ba784-3877-4627-bbd5-b65bdb734faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:58:40.703306 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.703289    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj\" (UID: \"e78ba784-3877-4627-bbd5-b65bdb734faa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:58:40.866250 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.866222    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:58:40.895964 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.895940    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerID="cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69" exitCode=6
Jul 17 21:58:40.896353 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.895981    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerDied","Data":"cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69"}
Jul 17 21:58:40.896353 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.896029    2546 scope.go:117] "RemoveContainer" containerID="f46a8551340fffd4f96a131125b53e7d3e744f647a5c3e972b9cc403eabef839"
Jul 17 21:58:40.896353 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.896251    2546 scope.go:117] "RemoveContainer" containerID="cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69"
Jul 17 21:58:40.896507 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:40.896475    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst_odh-ai-gateway-infra(bd344f19-d660-4878-ac04-3ac9b158bc8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a"
Jul 17 21:58:40.993955 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:40.993933    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:58:40.995292 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:58:40.995267    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode78ba784_3877_4627_bbd5_b65bdb734faa.slice/crio-76e0c36381812d26208b8042f857911e25b764230d4d6ad9d34ef78e9e0c8503 WatchSource:0}: Error finding container 76e0c36381812d26208b8042f857911e25b764230d4d6ad9d34ef78e9e0c8503: Status 404 returned error can't find the container with id 76e0c36381812d26208b8042f857911e25b764230d4d6ad9d34ef78e9e0c8503
Jul 17 21:58:41.902090 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:41.902004    2546 generic.go:356] "Generic (PLEG): container finished" podID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerID="d0573c33773211524abf6a364ccee099c4d1e4e751122490d7ce237cf705fec1" exitCode=6
Jul 17 21:58:41.902519 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:41.902089    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerDied","Data":"d0573c33773211524abf6a364ccee099c4d1e4e751122490d7ce237cf705fec1"}
Jul 17 21:58:41.902519 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:41.902141    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerStarted","Data":"76e0c36381812d26208b8042f857911e25b764230d4d6ad9d34ef78e9e0c8503"}
Jul 17 21:58:41.902519 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:41.902268    2546 scope.go:117] "RemoveContainer" containerID="d0573c33773211524abf6a364ccee099c4d1e4e751122490d7ce237cf705fec1"
Jul 17 21:58:41.903878 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:41.903859    2546 scope.go:117] "RemoveContainer" containerID="cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69"
Jul 17 21:58:41.904054 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:41.904035    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst_odh-ai-gateway-infra(bd344f19-d660-4878-ac04-3ac9b158bc8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a"
Jul 17 21:58:42.915083 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:42.915051    2546 generic.go:356] "Generic (PLEG): container finished" podID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerID="42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd" exitCode=6
Jul 17 21:58:42.915497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:42.915123    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerDied","Data":"42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd"}
Jul 17 21:58:42.915497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:42.915202    2546 scope.go:117] "RemoveContainer" containerID="d0573c33773211524abf6a364ccee099c4d1e4e751122490d7ce237cf705fec1"
Jul 17 21:58:42.915497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:42.915384    2546 scope.go:117] "RemoveContainer" containerID="42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd"
Jul 17 21:58:42.915633 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:42.915608    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj_odh-ai-gateway-infra(e78ba784-3877-4627-bbd5-b65bdb734faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa"
Jul 17 21:58:43.921145 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:43.921070    2546 scope.go:117] "RemoveContainer" containerID="42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd"
Jul 17 21:58:43.921501 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:43.921229    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj_odh-ai-gateway-infra(e78ba784-3877-4627-bbd5-b65bdb734faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa"
Jul 17 21:58:45.531601 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:45.531569    2546 scope.go:117] "RemoveContainer" containerID="bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd"
Jul 17 21:58:45.931311 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:45.931287    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerStarted","Data":"71942cc40fe3eb4010827d475f686e58a2661ecc5e62943cf69a960afa9f5261"}
Jul 17 21:58:45.949856 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:45.949811    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" podStartSLOduration=18.949795554 podStartE2EDuration="18.949795554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:45.948295726 +0000 UTC m=+2501.928666202" watchObservedRunningTime="2026-07-17 21:58:45.949795554 +0000 UTC m=+2501.930166069"
Jul 17 21:58:46.937483 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:46.937449    2546 generic.go:356] "Generic (PLEG): container finished" podID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerID="71942cc40fe3eb4010827d475f686e58a2661ecc5e62943cf69a960afa9f5261" exitCode=6
Jul 17 21:58:46.937956 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:46.937523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerDied","Data":"71942cc40fe3eb4010827d475f686e58a2661ecc5e62943cf69a960afa9f5261"}
Jul 17 21:58:46.937956 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:46.937582    2546 scope.go:117] "RemoveContainer" containerID="bf90480b5243eadba7f25daee62795c5a0f4f71bf7bd8a448d3b6862bdd053fd"
Jul 17 21:58:46.937956 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:46.937870    2546 scope.go:117] "RemoveContainer" containerID="71942cc40fe3eb4010827d475f686e58a2661ecc5e62943cf69a960afa9f5261"
Jul 17 21:58:46.938130 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:46.938048    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf_odh-ai-gateway-infra(ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9"
Jul 17 21:58:46.963659 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:46.963636    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:48.074685 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.074658    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:48.157032 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.156999    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca\") pod \"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9\" (UID: \"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9\") "
Jul 17 21:58:48.157378 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.157354    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" (UID: "ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:48.258055 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.258022    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:58:48.950523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.950439    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf" event={"ID":"ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9","Type":"ContainerDied","Data":"aa3020150ee4cacf662f70d7a1b2b0eca25494c7f39337143e494e28f24b71fe"}
Jul 17 21:58:48.950523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.950463    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"
Jul 17 21:58:48.950523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.950495    2546 scope.go:117] "RemoveContainer" containerID="71942cc40fe3eb4010827d475f686e58a2661ecc5e62943cf69a960afa9f5261"
Jul 17 21:58:48.966274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.966251    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:48.971184 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:48.971164    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kmcwf"]
Jul 17 21:58:50.536590 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:50.536556    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" path="/var/lib/kubelet/pods/ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9/volumes"
Jul 17 21:58:54.536694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:54.536655    2546 scope.go:117] "RemoveContainer" containerID="cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69"
Jul 17 21:58:54.982531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:54.982499    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerStarted","Data":"32765d292f621689f72c033aefd08b332b7a8f2ed1b9dc017d76332032ce9f9a"}
Jul 17 21:58:55.000695 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:55.000647    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" podStartSLOduration=17.000631941 podStartE2EDuration="17.000631941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:54.999474206 +0000 UTC m=+2510.979844684" watchObservedRunningTime="2026-07-17 21:58:55.000631941 +0000 UTC m=+2510.981002423"
Jul 17 21:58:55.989319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:55.989285    2546 generic.go:356] "Generic (PLEG): container finished" podID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerID="32765d292f621689f72c033aefd08b332b7a8f2ed1b9dc017d76332032ce9f9a" exitCode=6
Jul 17 21:58:55.989319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:55.989321    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerDied","Data":"32765d292f621689f72c033aefd08b332b7a8f2ed1b9dc017d76332032ce9f9a"}
Jul 17 21:58:55.990003 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:55.989369    2546 scope.go:117] "RemoveContainer" containerID="cd97ed3bbb11662f0152b5dc99eee6132af69664f4ba27bbadbb9cb253d71e69"
Jul 17 21:58:55.990003 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:55.989696    2546 scope.go:117] "RemoveContainer" containerID="32765d292f621689f72c033aefd08b332b7a8f2ed1b9dc017d76332032ce9f9a"
Jul 17 21:58:55.990003 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:55.989912    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst_odh-ai-gateway-infra(bd344f19-d660-4878-ac04-3ac9b158bc8a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a"
Jul 17 21:58:56.011465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:56.011443    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:57.126871 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:57.126843    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:57.229935 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:57.229904    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca\") pod \"bd344f19-d660-4878-ac04-3ac9b158bc8a\" (UID: \"bd344f19-d660-4878-ac04-3ac9b158bc8a\") "
Jul 17 21:58:57.230228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:57.230205    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd344f19-d660-4878-ac04-3ac9b158bc8a" (UID: "bd344f19-d660-4878-ac04-3ac9b158bc8a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:58:57.331438 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:57.331359    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd344f19-d660-4878-ac04-3ac9b158bc8a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:58:57.531226 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:57.531185    2546 scope.go:117] "RemoveContainer" containerID="42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd"
Jul 17 21:58:58.002070 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.002035    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerStarted","Data":"d0f254dcb792ae449674594312ea75d2bb56e4026f2b2275207231a6c2a08af9"}
Jul 17 21:58:58.003266 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.003242    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"
Jul 17 21:58:58.003266 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.003251    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst" event={"ID":"bd344f19-d660-4878-ac04-3ac9b158bc8a","Type":"ContainerDied","Data":"2c5d5858cc71a4018bd69879a31bb974e19173fe058c4f2ff0b9cbc1e67dd63d"}
Jul 17 21:58:58.003451 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.003303    2546 scope.go:117] "RemoveContainer" containerID="32765d292f621689f72c033aefd08b332b7a8f2ed1b9dc017d76332032ce9f9a"
Jul 17 21:58:58.018923 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.018879    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" podStartSLOduration=18.018865354 podStartE2EDuration="18.018865354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:58:58.017049426 +0000 UTC m=+2513.997419906" watchObservedRunningTime="2026-07-17 21:58:58.018865354 +0000 UTC m=+2513.999235831"
Jul 17 21:58:58.031212 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.031160    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:58.034521 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.034498    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-2kkst"]
Jul 17 21:58:58.535633 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:58.535583    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" path="/var/lib/kubelet/pods/bd344f19-d660-4878-ac04-3ac9b158bc8a/volumes"
Jul 17 21:58:59.010336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:59.010306    2546 generic.go:356] "Generic (PLEG): container finished" podID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerID="d0f254dcb792ae449674594312ea75d2bb56e4026f2b2275207231a6c2a08af9" exitCode=6
Jul 17 21:58:59.010531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:59.010377    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerDied","Data":"d0f254dcb792ae449674594312ea75d2bb56e4026f2b2275207231a6c2a08af9"}
Jul 17 21:58:59.010531 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:59.010435    2546 scope.go:117] "RemoveContainer" containerID="42fd701a9532fab227e72958e3e1059bba9d769264b0bcc8c00ac72a0cfd08fd"
Jul 17 21:58:59.010807 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:59.010785    2546 scope.go:117] "RemoveContainer" containerID="d0f254dcb792ae449674594312ea75d2bb56e4026f2b2275207231a6c2a08af9"
Jul 17 21:58:59.011035 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:58:59.011016    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj_odh-ai-gateway-infra(e78ba784-3877-4627-bbd5-b65bdb734faa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa"
Jul 17 21:58:59.030244 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:58:59.030221    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:59:00.148279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:00.148252    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:59:00.256782 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:00.256741    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca\") pod \"e78ba784-3877-4627-bbd5-b65bdb734faa\" (UID: \"e78ba784-3877-4627-bbd5-b65bdb734faa\") "
Jul 17 21:59:00.257103 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:00.257078    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e78ba784-3877-4627-bbd5-b65bdb734faa" (UID: "e78ba784-3877-4627-bbd5-b65bdb734faa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:00.357321 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:00.357234    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e78ba784-3877-4627-bbd5-b65bdb734faa-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:59:01.023859 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:01.023819    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj" event={"ID":"e78ba784-3877-4627-bbd5-b65bdb734faa","Type":"ContainerDied","Data":"76e0c36381812d26208b8042f857911e25b764230d4d6ad9d34ef78e9e0c8503"}
Jul 17 21:59:01.023859 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:01.023840    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"
Jul 17 21:59:01.024119 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:01.023890    2546 scope.go:117] "RemoveContainer" containerID="d0f254dcb792ae449674594312ea75d2bb56e4026f2b2275207231a6c2a08af9"
Jul 17 21:59:01.039223 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:01.039198    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:59:01.042898 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:01.042877    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq8hj"]
Jul 17 21:59:02.535910 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:02.535877    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" path="/var/lib/kubelet/pods/e78ba784-3877-4627-bbd5-b65bdb734faa/volumes"
Jul 17 21:59:15.325991 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.325939    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326228    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326238    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326248    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326254    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326263    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326269    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326277    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326282    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326289    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326294    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326306    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326310    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326316    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326320    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326376    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326383    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326389    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326395    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326399    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326406    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326411    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.326462 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.326416    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:15.329020 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.329004    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:15.330986 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.330970    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 21:59:15.337151 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.337121    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:15.474621 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.474589    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b\" (UID: \"d013762e-ea6f-4e88-8cd9-081a861accac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:15.575079 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.575043    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b\" (UID: \"d013762e-ea6f-4e88-8cd9-081a861accac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:15.575643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.575624    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b\" (UID: \"d013762e-ea6f-4e88-8cd9-081a861accac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:15.640224 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.640154    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:15.775026 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:15.775001    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:15.776699 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:15.776674    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd013762e_ea6f_4e88_8cd9_081a861accac.slice/crio-6e73bf80b1738df3bdcb052ecb43a425e8fc14088dc60a58cd7d87aeeff0e40d WatchSource:0}: Error finding container 6e73bf80b1738df3bdcb052ecb43a425e8fc14088dc60a58cd7d87aeeff0e40d: Status 404 returned error can't find the container with id 6e73bf80b1738df3bdcb052ecb43a425e8fc14088dc60a58cd7d87aeeff0e40d
Jul 17 21:59:16.095457 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:16.095428    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerStarted","Data":"b688da62dcd8d4caa93d7a604bffaafcf725d14ec923081ebc44d7f0de397821"}
Jul 17 21:59:16.095610 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:16.095480    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerStarted","Data":"6e73bf80b1738df3bdcb052ecb43a425e8fc14088dc60a58cd7d87aeeff0e40d"}
Jul 17 21:59:16.113259 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:16.113215    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" podStartSLOduration=1.113201542 podStartE2EDuration="1.113201542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:16.110551551 +0000 UTC m=+2532.090922022" watchObservedRunningTime="2026-07-17 21:59:16.113201542 +0000 UTC m=+2532.093572019"
Jul 17 21:59:17.101294 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:17.101258    2546 generic.go:356] "Generic (PLEG): container finished" podID="d013762e-ea6f-4e88-8cd9-081a861accac" containerID="b688da62dcd8d4caa93d7a604bffaafcf725d14ec923081ebc44d7f0de397821" exitCode=6
Jul 17 21:59:17.101693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:17.101351    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerDied","Data":"b688da62dcd8d4caa93d7a604bffaafcf725d14ec923081ebc44d7f0de397821"}
Jul 17 21:59:17.101693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:17.101497    2546 scope.go:117] "RemoveContainer" containerID="b688da62dcd8d4caa93d7a604bffaafcf725d14ec923081ebc44d7f0de397821"
Jul 17 21:59:18.108390 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:18.108353    2546 generic.go:356] "Generic (PLEG): container finished" podID="d013762e-ea6f-4e88-8cd9-081a861accac" containerID="fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f" exitCode=6
Jul 17 21:59:18.108783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:18.108444    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerDied","Data":"fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f"}
Jul 17 21:59:18.108783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:18.108501    2546 scope.go:117] "RemoveContainer" containerID="b688da62dcd8d4caa93d7a604bffaafcf725d14ec923081ebc44d7f0de397821"
Jul 17 21:59:18.108783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:18.108588    2546 scope.go:117] "RemoveContainer" containerID="fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f"
Jul 17 21:59:18.108965 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:18.108838    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b_odh-ai-gateway-infra(d013762e-ea6f-4e88-8cd9-081a861accac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" podUID="d013762e-ea6f-4e88-8cd9-081a861accac"
Jul 17 21:59:19.115054 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:19.115031    2546 scope.go:117] "RemoveContainer" containerID="fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f"
Jul 17 21:59:19.115427 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:19.115190    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b_odh-ai-gateway-infra(d013762e-ea6f-4e88-8cd9-081a861accac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" podUID="d013762e-ea6f-4e88-8cd9-081a861accac"
Jul 17 21:59:23.123002 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.122972    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 21:59:23.123400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.123273    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:23.123400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.123284    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd344f19-d660-4878-ac04-3ac9b158bc8a" containerName="revoke-keys"
Jul 17 21:59:23.123400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.123303    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:23.123400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.123310    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e78ba784-3877-4627-bbd5-b65bdb734faa" containerName="revoke-keys"
Jul 17 21:59:23.123400 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.123355    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea2d6c6b-7e31-4f8a-8cbb-4fbc7fde87d9" containerName="revoke-keys"
Jul 17 21:59:23.127476 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.127453    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.130747 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.130711    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-6e20ff19-openshift-default-dockercfg-rnwwt\""
Jul 17 21:59:23.145277 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.145252    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 21:59:23.233807 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.233776    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.233999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.233889    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.233999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.233930    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.233999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.233952    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.233999 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.233975    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.234176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.234055    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.234176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.234103    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.234176 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.234154    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69tmq\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.234307 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.234205    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335273 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335236    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335313    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335341    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335367    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335397    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69tmq\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335452 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335433    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335465    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335537    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335826 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335814    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.335987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.335965    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.336101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.336074    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.336228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.336139    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.336228 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.336191    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.338011 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.337983    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.338179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.338163    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.345428 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.345403    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69tmq\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.345551 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.345513    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token\") pod \"e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") " pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.437511 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.437437    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:23.575591 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.575568    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 21:59:23.577469 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:23.577433    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781fb554_dea3_4b6e_a52d_0ebf80de2124.slice/crio-200e192424594870584ab1d8d6d92e270b7d40c79cb8eb84ade74e34138d81d7 WatchSource:0}: Error finding container 200e192424594870584ab1d8d6d92e270b7d40c79cb8eb84ade74e34138d81d7: Status 404 returned error can't find the container with id 200e192424594870584ab1d8d6d92e270b7d40c79cb8eb84ade74e34138d81d7
Jul 17 21:59:23.579489 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.579461    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:23.579557 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.579529    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:23.579603 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:23.579558    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:24.140014 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:24.139981    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerStarted","Data":"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"}
Jul 17 21:59:24.140497 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:24.140036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerStarted","Data":"200e192424594870584ab1d8d6d92e270b7d40c79cb8eb84ade74e34138d81d7"}
Jul 17 21:59:24.159900 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:24.159859    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podStartSLOduration=1.159843185 podStartE2EDuration="1.159843185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 21:59:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 21:59:24.158860436 +0000 UTC m=+2540.139230915" watchObservedRunningTime="2026-07-17 21:59:24.159843185 +0000 UTC m=+2540.140213661"
Jul 17 21:59:24.437935 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:24.437842    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:24.442917 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:24.442892    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:25.144894 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:25.144866    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:25.145893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:25.145873    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:27.781279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.781234    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:27.784809 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.784787    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:27.794702 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.794669    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:27.878086 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.878052    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv\" (UID: \"38e2f34b-a292-4d2b-8614-1b14131525ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:27.979346 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.979313    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv\" (UID: \"38e2f34b-a292-4d2b-8614-1b14131525ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:27.979953 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:27.979933    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv\" (UID: \"38e2f34b-a292-4d2b-8614-1b14131525ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:28.095144 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.095056    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:28.233394 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.233353    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:28.236199 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:28.236167    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38e2f34b_a292_4d2b_8614_1b14131525ba.slice/crio-2137c3a57dd422df9797aff8d404e1b9d8119de71a8d6db0434ffb1343c167ad WatchSource:0}: Error finding container 2137c3a57dd422df9797aff8d404e1b9d8119de71a8d6db0434ffb1343c167ad: Status 404 returned error can't find the container with id 2137c3a57dd422df9797aff8d404e1b9d8119de71a8d6db0434ffb1343c167ad
Jul 17 21:59:28.764109 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.764075    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:28.767205 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.767190    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:28.775363 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.775331    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:28.887179 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.887154    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz\" (UID: \"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:28.988239 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.988206    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz\" (UID: \"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:28.988842 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:28.988824    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz\" (UID: \"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:29.077631 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.077554    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:29.180745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.178757    2546 generic.go:356] "Generic (PLEG): container finished" podID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerID="acf6d116db63f138aa654ad5f932f7c980448a19dcabaf40f7c7555dc616b31e" exitCode=6
Jul 17 21:59:29.180745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.178846    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" event={"ID":"38e2f34b-a292-4d2b-8614-1b14131525ba","Type":"ContainerDied","Data":"acf6d116db63f138aa654ad5f932f7c980448a19dcabaf40f7c7555dc616b31e"}
Jul 17 21:59:29.180745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.178897    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" event={"ID":"38e2f34b-a292-4d2b-8614-1b14131525ba","Type":"ContainerStarted","Data":"2137c3a57dd422df9797aff8d404e1b9d8119de71a8d6db0434ffb1343c167ad"}
Jul 17 21:59:29.180745 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.179193    2546 scope.go:117] "RemoveContainer" containerID="acf6d116db63f138aa654ad5f932f7c980448a19dcabaf40f7c7555dc616b31e"
Jul 17 21:59:29.238693 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:29.238671    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:29.241058 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:29.241038    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57e66de2_c4f6_45c8_b6ec_2783f8e1c9b8.slice/crio-6012942cc97fedbaf1fcefb9f9d80f332cd2a41e187394e173d964b82497e9d9 WatchSource:0}: Error finding container 6012942cc97fedbaf1fcefb9f9d80f332cd2a41e187394e173d964b82497e9d9: Status 404 returned error can't find the container with id 6012942cc97fedbaf1fcefb9f9d80f332cd2a41e187394e173d964b82497e9d9
Jul 17 21:59:30.185796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.185749    2546 generic.go:356] "Generic (PLEG): container finished" podID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerID="9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a" exitCode=6
Jul 17 21:59:30.186274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.185818    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" event={"ID":"38e2f34b-a292-4d2b-8614-1b14131525ba","Type":"ContainerDied","Data":"9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a"}
Jul 17 21:59:30.186274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.185886    2546 scope.go:117] "RemoveContainer" containerID="acf6d116db63f138aa654ad5f932f7c980448a19dcabaf40f7c7555dc616b31e"
Jul 17 21:59:30.186274 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.186008    2546 scope.go:117] "RemoveContainer" containerID="9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a"
Jul 17 21:59:30.186274 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:30.186238    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv_odh-ai-gateway-infra(38e2f34b-a292-4d2b-8614-1b14131525ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba"
Jul 17 21:59:30.187494 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.187471    2546 generic.go:356] "Generic (PLEG): container finished" podID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerID="fa8a3d50b69ce6d9e2681ebdd3fa317c8852cb9c7b38b3953b8e6df7aa8b60c5" exitCode=6
Jul 17 21:59:30.187605 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.187556    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" event={"ID":"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8","Type":"ContainerDied","Data":"fa8a3d50b69ce6d9e2681ebdd3fa317c8852cb9c7b38b3953b8e6df7aa8b60c5"}
Jul 17 21:59:30.187680 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.187604    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" event={"ID":"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8","Type":"ContainerStarted","Data":"6012942cc97fedbaf1fcefb9f9d80f332cd2a41e187394e173d964b82497e9d9"}
Jul 17 21:59:30.187766 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.187750    2546 scope.go:117] "RemoveContainer" containerID="fa8a3d50b69ce6d9e2681ebdd3fa317c8852cb9c7b38b3953b8e6df7aa8b60c5"
Jul 17 21:59:30.531325 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:30.531289    2546 scope.go:117] "RemoveContainer" containerID="fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f"
Jul 17 21:59:31.195346 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.195255    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerStarted","Data":"354d685aab523a6d54d7deb9eb6e1c24e5391febd5fbf26c1d9c60865308d46e"}
Jul 17 21:59:31.197005 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.196987    2546 scope.go:117] "RemoveContainer" containerID="9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a"
Jul 17 21:59:31.197181 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:31.197161    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv_odh-ai-gateway-infra(38e2f34b-a292-4d2b-8614-1b14131525ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba"
Jul 17 21:59:31.198544 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.198517    2546 generic.go:356] "Generic (PLEG): container finished" podID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerID="6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1" exitCode=6
Jul 17 21:59:31.198643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.198591    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" event={"ID":"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8","Type":"ContainerDied","Data":"6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1"}
Jul 17 21:59:31.198643 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.198635    2546 scope.go:117] "RemoveContainer" containerID="fa8a3d50b69ce6d9e2681ebdd3fa317c8852cb9c7b38b3953b8e6df7aa8b60c5"
Jul 17 21:59:31.198857 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:31.198841    2546 scope.go:117] "RemoveContainer" containerID="6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1"
Jul 17 21:59:31.199041 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:31.199020    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz_odh-ai-gateway-infra(57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8"
Jul 17 21:59:32.205279 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.205242    2546 generic.go:356] "Generic (PLEG): container finished" podID="d013762e-ea6f-4e88-8cd9-081a861accac" containerID="354d685aab523a6d54d7deb9eb6e1c24e5391febd5fbf26c1d9c60865308d46e" exitCode=6
Jul 17 21:59:32.205796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.205308    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerDied","Data":"354d685aab523a6d54d7deb9eb6e1c24e5391febd5fbf26c1d9c60865308d46e"}
Jul 17 21:59:32.205796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.205383    2546 scope.go:117] "RemoveContainer" containerID="fc7715d4769e20fa6cce3682051b651f67393659c9c7bbd771a840223876e68f"
Jul 17 21:59:32.205796 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.205703    2546 scope.go:117] "RemoveContainer" containerID="354d685aab523a6d54d7deb9eb6e1c24e5391febd5fbf26c1d9c60865308d46e"
Jul 17 21:59:32.205982 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:32.205959    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b_odh-ai-gateway-infra(d013762e-ea6f-4e88-8cd9-081a861accac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" podUID="d013762e-ea6f-4e88-8cd9-081a861accac"
Jul 17 21:59:32.207163 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.207144    2546 scope.go:117] "RemoveContainer" containerID="6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1"
Jul 17 21:59:32.207322 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:32.207305    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz_odh-ai-gateway-infra(57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8"
Jul 17 21:59:32.237089 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:32.237058    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:33.377746 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:33.372283    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:33.530835 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:33.530807    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca\") pod \"d013762e-ea6f-4e88-8cd9-081a861accac\" (UID: \"d013762e-ea6f-4e88-8cd9-081a861accac\") "
Jul 17 21:59:33.531101 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:33.531079    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d013762e-ea6f-4e88-8cd9-081a861accac" (UID: "d013762e-ea6f-4e88-8cd9-081a861accac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:33.632423 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:33.632390    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d013762e-ea6f-4e88-8cd9-081a861accac-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:59:34.220517 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.220490    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"
Jul 17 21:59:34.220687 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.220534    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b" event={"ID":"d013762e-ea6f-4e88-8cd9-081a861accac","Type":"ContainerDied","Data":"6e73bf80b1738df3bdcb052ecb43a425e8fc14088dc60a58cd7d87aeeff0e40d"}
Jul 17 21:59:34.220687 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.220592    2546 scope.go:117] "RemoveContainer" containerID="354d685aab523a6d54d7deb9eb6e1c24e5391febd5fbf26c1d9c60865308d46e"
Jul 17 21:59:34.245792 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.245766    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:34.248829 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.248796    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-bx89b"]
Jul 17 21:59:34.536686 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:34.536657    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" path="/var/lib/kubelet/pods/d013762e-ea6f-4e88-8cd9-081a861accac/volumes"
Jul 17 21:59:37.713975 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.713936    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"]
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714252    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714263    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714278    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714284    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714296    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714302    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714360    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.714404 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.714366    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:37.718609 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.718586    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:37.727388 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.727357    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"]
Jul 17 21:59:37.764429 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.764395    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph\" (UID: \"7bbbc586-57f2-4f36-82dd-d5a0694b37b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:37.865644 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.865600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph\" (UID: \"7bbbc586-57f2-4f36-82dd-d5a0694b37b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:37.866465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:37.866439    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph\" (UID: \"7bbbc586-57f2-4f36-82dd-d5a0694b37b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:38.031465 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:38.031435    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:38.173463 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:38.173439    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"]
Jul 17 21:59:38.176129 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:38.176102    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bbbc586_57f2_4f36_82dd_d5a0694b37b0.slice/crio-7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb WatchSource:0}: Error finding container 7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb: Status 404 returned error can't find the container with id 7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb
Jul 17 21:59:38.243231 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:38.243205    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph" event={"ID":"7bbbc586-57f2-4f36-82dd-d5a0694b37b0","Type":"ContainerStarted","Data":"7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb"}
Jul 17 21:59:39.250209 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:39.250181    2546 generic.go:356] "Generic (PLEG): container finished" podID="7bbbc586-57f2-4f36-82dd-d5a0694b37b0" containerID="2380c3b23b8389391ef4f21e49cd951432f8ce67cc8d7725f7fcfc89e0c89ce2" exitCode=0
Jul 17 21:59:39.251543 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:39.250272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph" event={"ID":"7bbbc586-57f2-4f36-82dd-d5a0694b37b0","Type":"ContainerDied","Data":"2380c3b23b8389391ef4f21e49cd951432f8ce67cc8d7725f7fcfc89e0c89ce2"}
Jul 17 21:59:40.146366 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.146329    2546 patch_prober.go:28] interesting pod/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Jul 17 21:59:40.146530 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.146381    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Jul 17 21:59:40.257189 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.257160    2546 generic.go:356] "Generic (PLEG): container finished" podID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerID="2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5" exitCode=0
Jul 17 21:59:40.257583 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.257234    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerDied","Data":"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"}
Jul 17 21:59:40.257998 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.257692    2546 scope.go:117] "RemoveContainer" containerID="2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"
Jul 17 21:59:40.258821 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.258790    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:40.258925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.258862    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:40.258925 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.258895    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 21:59:40.388199 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.388176    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:40.489676 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.489656    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca\") pod \"7bbbc586-57f2-4f36-82dd-d5a0694b37b0\" (UID: \"7bbbc586-57f2-4f36-82dd-d5a0694b37b0\") "
Jul 17 21:59:40.490006 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.489988    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7bbbc586-57f2-4f36-82dd-d5a0694b37b0" (UID: "7bbbc586-57f2-4f36-82dd-d5a0694b37b0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:40.591099 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.591065    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7bbbc586-57f2-4f36-82dd-d5a0694b37b0-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:59:40.599588 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.599562    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 21:59:40.599902 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.599890    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7bbbc586-57f2-4f36-82dd-d5a0694b37b0" containerName="revoke-keys"
Jul 17 21:59:40.599950 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.599904    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7bbbc586-57f2-4f36-82dd-d5a0694b37b0" containerName="revoke-keys"
Jul 17 21:59:40.599985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.599949    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7bbbc586-57f2-4f36-82dd-d5a0694b37b0" containerName="revoke-keys"
Jul 17 21:59:40.599985 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.599961    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d013762e-ea6f-4e88-8cd9-081a861accac" containerName="revoke-keys"
Jul 17 21:59:40.602673 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.602657    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 21:59:40.611043 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.611021    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 21:59:40.691976 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.691953    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb\" (UID: \"356116b1-208b-4c92-9d90-a03937e3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 21:59:40.793147 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.793067    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb\" (UID: \"356116b1-208b-4c92-9d90-a03937e3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 21:59:40.793647 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.793629    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb\" (UID: \"356116b1-208b-4c92-9d90-a03937e3213d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 21:59:40.913485 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:40.913453    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 21:59:41.084685 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.084658    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 21:59:41.087949 ip-10-0-135-220 kubenswrapper[2546]: W0717 21:59:41.087906    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod356116b1_208b_4c92_9d90_a03937e3213d.slice/crio-504290e0d65dd5e57187cd156b5dc05aa8be7a95743e8529559b3fe040a27951 WatchSource:0}: Error finding container 504290e0d65dd5e57187cd156b5dc05aa8be7a95743e8529559b3fe040a27951: Status 404 returned error can't find the container with id 504290e0d65dd5e57187cd156b5dc05aa8be7a95743e8529559b3fe040a27951
Jul 17 21:59:41.288149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.287140    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph" event={"ID":"7bbbc586-57f2-4f36-82dd-d5a0694b37b0","Type":"ContainerDied","Data":"7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb"}
Jul 17 21:59:41.288149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.287209    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7b560d1c1c902740c9c9ec22a3ed1948f4d3c8f481389f36b810e71575375cbb"
Jul 17 21:59:41.288149 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.287291    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph"
Jul 17 21:59:41.290357 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.290330    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerStarted","Data":"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"}
Jul 17 21:59:41.291808 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.291787    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" event={"ID":"356116b1-208b-4c92-9d90-a03937e3213d","Type":"ContainerStarted","Data":"504290e0d65dd5e57187cd156b5dc05aa8be7a95743e8529559b3fe040a27951"}
Jul 17 21:59:41.437972 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.437944    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:41.438160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.438139    2546 patch_prober.go:28] interesting pod/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Jul 17 21:59:41.438237 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:41.438192    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Jul 17 21:59:42.298131 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:42.298040    2546 generic.go:356] "Generic (PLEG): container finished" podID="356116b1-208b-4c92-9d90-a03937e3213d" containerID="dd1ae1c66f504cb7fb48785ca38868dcb987b443eaebe6b0cd8c89a0a1c1e136" exitCode=6
Jul 17 21:59:42.298523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:42.298126    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" event={"ID":"356116b1-208b-4c92-9d90-a03937e3213d","Type":"ContainerDied","Data":"dd1ae1c66f504cb7fb48785ca38868dcb987b443eaebe6b0cd8c89a0a1c1e136"}
Jul 17 21:59:42.298523 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:42.298256    2546 scope.go:117] "RemoveContainer" containerID="dd1ae1c66f504cb7fb48785ca38868dcb987b443eaebe6b0cd8c89a0a1c1e136"
Jul 17 21:59:42.438770 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:42.438744    2546 patch_prober.go:28] interesting pod/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Jul 17 21:59:42.438905 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:42.438789    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Jul 17 21:59:43.305112 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.305077    2546 generic.go:356] "Generic (PLEG): container finished" podID="356116b1-208b-4c92-9d90-a03937e3213d" containerID="31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad" exitCode=6
Jul 17 21:59:43.305542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.305158    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" event={"ID":"356116b1-208b-4c92-9d90-a03937e3213d","Type":"ContainerDied","Data":"31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad"}
Jul 17 21:59:43.305542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.305213    2546 scope.go:117] "RemoveContainer" containerID="dd1ae1c66f504cb7fb48785ca38868dcb987b443eaebe6b0cd8c89a0a1c1e136"
Jul 17 21:59:43.305542 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.305491    2546 scope.go:117] "RemoveContainer" containerID="31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad"
Jul 17 21:59:43.305702 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:43.305681    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb_odh-ai-gateway-infra(356116b1-208b-4c92-9d90-a03937e3213d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" podUID="356116b1-208b-4c92-9d90-a03937e3213d"
Jul 17 21:59:43.438155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.438075    2546 patch_prober.go:28] interesting pod/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Jul 17 21:59:43.438155 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:43.438140    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Jul 17 21:59:44.312089 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:44.312057    2546 scope.go:117] "RemoveContainer" containerID="31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad"
Jul 17 21:59:44.312565 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:44.312282    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb_odh-ai-gateway-infra(356116b1-208b-4c92-9d90-a03937e3213d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" podUID="356116b1-208b-4c92-9d90-a03937e3213d"
Jul 17 21:59:44.438750 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:44.438703    2546 patch_prober.go:28] interesting pod/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused" start-of-body=
Jul 17 21:59:44.438958 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:44.438772    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.93:15021/healthz/ready\": dial tcp 10.133.0.93:15021: connect: connection refused"
Jul 17 21:59:45.441771 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:45.441735    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:45.442136 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:45.441982    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:45.442783 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:45.442763    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 21:59:45.531931 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:45.531901    2546 scope.go:117] "RemoveContainer" containerID="9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a"
Jul 17 21:59:46.323025 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:46.322998    2546 generic.go:356] "Generic (PLEG): container finished" podID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerID="793e49cb8b4c27ecb414aefc88b4c67dafe7e02780f3aa8ece1b5767a1bc1a35" exitCode=6
Jul 17 21:59:46.323221 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:46.323070    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" event={"ID":"38e2f34b-a292-4d2b-8614-1b14131525ba","Type":"ContainerDied","Data":"793e49cb8b4c27ecb414aefc88b4c67dafe7e02780f3aa8ece1b5767a1bc1a35"}
Jul 17 21:59:46.323221 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:46.323129    2546 scope.go:117] "RemoveContainer" containerID="9f0f922f7b962b2ca302a830b47353b3d1ba35255be92f122d047875ed84bd9a"
Jul 17 21:59:46.323524 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:46.323501    2546 scope.go:117] "RemoveContainer" containerID="793e49cb8b4c27ecb414aefc88b4c67dafe7e02780f3aa8ece1b5767a1bc1a35"
Jul 17 21:59:46.323702 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:46.323680    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv_odh-ai-gateway-infra(38e2f34b-a292-4d2b-8614-1b14131525ba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba"
Jul 17 21:59:46.531521 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:46.531501    2546 scope.go:117] "RemoveContainer" containerID="6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1"
Jul 17 21:59:47.336676 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.336642    2546 generic.go:356] "Generic (PLEG): container finished" podID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerID="76915597740d2297dd132612a668f42cc238f2ec65624e911b2d53e6a537e932" exitCode=6
Jul 17 21:59:47.336801 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.336716    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" event={"ID":"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8","Type":"ContainerDied","Data":"76915597740d2297dd132612a668f42cc238f2ec65624e911b2d53e6a537e932"}
Jul 17 21:59:47.336849 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.336816    2546 scope.go:117] "RemoveContainer" containerID="6b9c923f1a5b03657e39e51677296516ed85ac9fcbd2aed958da873871e7a7f1"
Jul 17 21:59:47.337067 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.337047    2546 scope.go:117] "RemoveContainer" containerID="76915597740d2297dd132612a668f42cc238f2ec65624e911b2d53e6a537e932"
Jul 17 21:59:47.337262 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:47.337242    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz_odh-ai-gateway-infra(57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8"
Jul 17 21:59:47.347386 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.347364    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:47.473679 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.473659    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:47.544613 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.544551    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca\") pod \"38e2f34b-a292-4d2b-8614-1b14131525ba\" (UID: \"38e2f34b-a292-4d2b-8614-1b14131525ba\") "
Jul 17 21:59:47.544942 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.544849    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38e2f34b-a292-4d2b-8614-1b14131525ba" (UID: "38e2f34b-a292-4d2b-8614-1b14131525ba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:47.646245 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:47.646218    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38e2f34b-a292-4d2b-8614-1b14131525ba-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:59:48.342377 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.342355    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"
Jul 17 21:59:48.342377 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.342356    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv" event={"ID":"38e2f34b-a292-4d2b-8614-1b14131525ba","Type":"ContainerDied","Data":"2137c3a57dd422df9797aff8d404e1b9d8119de71a8d6db0434ffb1343c167ad"}
Jul 17 21:59:48.342653 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.342426    2546 scope.go:117] "RemoveContainer" containerID="793e49cb8b4c27ecb414aefc88b4c67dafe7e02780f3aa8ece1b5767a1bc1a35"
Jul 17 21:59:48.362987 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.362962    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:48.370120 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.370093    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:48.379861 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.379837    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-chnbv"]
Jul 17 21:59:48.498832 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.498809    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:48.536617 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.536587    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" path="/var/lib/kubelet/pods/38e2f34b-a292-4d2b-8614-1b14131525ba/volumes"
Jul 17 21:59:48.554421 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.554401    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca\") pod \"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8\" (UID: \"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8\") "
Jul 17 21:59:48.554767 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.554694    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" (UID: "57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 21:59:48.655397 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:48.655334    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 21:59:49.352122 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:49.352093    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"
Jul 17 21:59:49.352336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:49.352142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz" event={"ID":"57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8","Type":"ContainerDied","Data":"6012942cc97fedbaf1fcefb9f9d80f332cd2a41e187394e173d964b82497e9d9"}
Jul 17 21:59:49.352336 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:49.352217    2546 scope.go:117] "RemoveContainer" containerID="76915597740d2297dd132612a668f42cc238f2ec65624e911b2d53e6a537e932"
Jul 17 21:59:49.389484 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:49.389445    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:49.394319 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:49.394295    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-bwpkz"]
Jul 17 21:59:50.536160 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:50.536128    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" path="/var/lib/kubelet/pods/57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8/volumes"
Jul 17 21:59:58.531886 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:58.531852    2546 scope.go:117] "RemoveContainer" containerID="31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad"
Jul 17 21:59:59.413694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:59.413605    2546 generic.go:356] "Generic (PLEG): container finished" podID="356116b1-208b-4c92-9d90-a03937e3213d" containerID="5d61030a93e830a0bbb00e2f6a89c4f6e22530fa010542ce20135d3a97af63c9" exitCode=6
Jul 17 21:59:59.413694 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:59.413672    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" event={"ID":"356116b1-208b-4c92-9d90-a03937e3213d","Type":"ContainerDied","Data":"5d61030a93e830a0bbb00e2f6a89c4f6e22530fa010542ce20135d3a97af63c9"}
Jul 17 21:59:59.413893 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:59.413782    2546 scope.go:117] "RemoveContainer" containerID="31abfe6254e0415ecc81f1c22a45d7f13edbca3d78edb58fab3c70533ccdb1ad"
Jul 17 21:59:59.414051 ip-10-0-135-220 kubenswrapper[2546]: I0717 21:59:59.414039    2546 scope.go:117] "RemoveContainer" containerID="5d61030a93e830a0bbb00e2f6a89c4f6e22530fa010542ce20135d3a97af63c9"
Jul 17 21:59:59.414252 ip-10-0-135-220 kubenswrapper[2546]: E0717 21:59:59.414235    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb_odh-ai-gateway-infra(356116b1-208b-4c92-9d90-a03937e3213d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" podUID="356116b1-208b-4c92-9d90-a03937e3213d"
Jul 17 22:00:00.142884 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.142848    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"]
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143149    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143160    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143169    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143175    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143183    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143189    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143197    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143202    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143215    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143220    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143225    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143229    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143279    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143286    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143293    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:00.143309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.143299    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:00.147424 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.147407    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"
Jul 17 22:00:00.155045 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.155006    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"]
Jul 17 22:00:00.158447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.158424    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"
Jul 17 22:00:00.444506 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.444435    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 22:00:00.500891 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.500861    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"]
Jul 17 22:00:00.507392 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:00.507359    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode240e5d5_feee_47df_a2b0_e8fb59ada5a5.slice/crio-68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d WatchSource:0}: Error finding container 68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d: Status 404 returned error can't find the container with id 68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d
Jul 17 22:00:00.588067 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.588045    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 22:00:00.646216 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.646187    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca\") pod \"356116b1-208b-4c92-9d90-a03937e3213d\" (UID: \"356116b1-208b-4c92-9d90-a03937e3213d\") "
Jul 17 22:00:00.646608 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.646579    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "356116b1-208b-4c92-9d90-a03937e3213d" (UID: "356116b1-208b-4c92-9d90-a03937e3213d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:00.747282 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:00.747262    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/356116b1-208b-4c92-9d90-a03937e3213d-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:01.426379 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.426352    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"
Jul 17 22:00:01.426781 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.426369    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb" event={"ID":"356116b1-208b-4c92-9d90-a03937e3213d","Type":"ContainerDied","Data":"504290e0d65dd5e57187cd156b5dc05aa8be7a95743e8529559b3fe040a27951"}
Jul 17 22:00:01.426781 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.426462    2546 scope.go:117] "RemoveContainer" containerID="5d61030a93e830a0bbb00e2f6a89c4f6e22530fa010542ce20135d3a97af63c9"
Jul 17 22:00:01.427886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.427864    2546 generic.go:356] "Generic (PLEG): container finished" podID="e240e5d5-feee-47df-a2b0-e8fb59ada5a5" containerID="7cf2d40cfbaea182a72b1708ede01239f17f11bc269aca9497f15a36311016bc" exitCode=0
Jul 17 22:00:01.427958 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.427930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp" event={"ID":"e240e5d5-feee-47df-a2b0-e8fb59ada5a5","Type":"ContainerDied","Data":"7cf2d40cfbaea182a72b1708ede01239f17f11bc269aca9497f15a36311016bc"}
Jul 17 22:00:01.428005 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.427960    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp" event={"ID":"e240e5d5-feee-47df-a2b0-e8fb59ada5a5","Type":"ContainerStarted","Data":"68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d"}
Jul 17 22:00:01.462697 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.462671    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 22:00:01.466147 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:01.466128    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-9kllb"]
Jul 17 22:00:02.537675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:02.537635    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="356116b1-208b-4c92-9d90-a03937e3213d" path="/var/lib/kubelet/pods/356116b1-208b-4c92-9d90-a03937e3213d/volumes"
Jul 17 22:00:02.560801 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:02.560780    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"
Jul 17 22:00:03.440917 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:03.440883    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp" event={"ID":"e240e5d5-feee-47df-a2b0-e8fb59ada5a5","Type":"ContainerDied","Data":"68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d"}
Jul 17 22:00:03.440917 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:03.440911    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738760-rn7hp"
Jul 17 22:00:03.441125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:03.440937    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68da68792c257152035700afe0d663103b7232d903f9cb44bc05d0a1b314173d"
Jul 17 22:00:03.542899 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:03.542865    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 22:00:03.543292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:03.543109    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy" containerID="cri-o://3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713" gracePeriod=30
Jul 17 22:00:05.364833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.364802    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365102    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365112    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365124    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365129    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365153    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e240e5d5-feee-47df-a2b0-e8fb59ada5a5" containerName="cleanup"
Jul 17 22:00:05.365186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365159    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e240e5d5-feee-47df-a2b0-e8fb59ada5a5" containerName="cleanup"
Jul 17 22:00:05.365390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365203    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.365390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365210    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e240e5d5-feee-47df-a2b0-e8fb59ada5a5" containerName="cleanup"
Jul 17 22:00:05.365390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365216    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="57e66de2-c4f6-45c8-b6ec-2783f8e1c9b8" containerName="revoke-keys"
Jul 17 22:00:05.365390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365224    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="38e2f34b-a292-4d2b-8614-1b14131525ba" containerName="revoke-keys"
Jul 17 22:00:05.365390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.365229    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:05.369607 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.369590    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:05.372081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.372052    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:00:05.376137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.376115    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:05.485629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.485598    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b\" (UID: \"6dc1e0c9-bb05-442f-808f-2aa0314cdff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:05.585948 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.585916    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b\" (UID: \"6dc1e0c9-bb05-442f-808f-2aa0314cdff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:05.586512 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.586492    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b\" (UID: \"6dc1e0c9-bb05-442f-808f-2aa0314cdff0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:05.681119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.681048    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:05.815386 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:05.815366    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:05.816991 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:05.816965    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dc1e0c9_bb05_442f_808f_2aa0314cdff0.slice/crio-087540aac7edeae5d4718212d8cf895e1d5a28b330dd9eb0a60771a7debf82db WatchSource:0}: Error finding container 087540aac7edeae5d4718212d8cf895e1d5a28b330dd9eb0a60771a7debf82db: Status 404 returned error can't find the container with id 087540aac7edeae5d4718212d8cf895e1d5a28b330dd9eb0a60771a7debf82db
Jul 17 22:00:06.456574 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:06.456546    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerID="7e6428fedd73892d228bad589314745d674f1d5a5b37313a371cae55ed057b1f" exitCode=6
Jul 17 22:00:06.456885 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:06.456598    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" event={"ID":"6dc1e0c9-bb05-442f-808f-2aa0314cdff0","Type":"ContainerDied","Data":"7e6428fedd73892d228bad589314745d674f1d5a5b37313a371cae55ed057b1f"}
Jul 17 22:00:06.456885 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:06.456644    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" event={"ID":"6dc1e0c9-bb05-442f-808f-2aa0314cdff0","Type":"ContainerStarted","Data":"087540aac7edeae5d4718212d8cf895e1d5a28b330dd9eb0a60771a7debf82db"}
Jul 17 22:00:06.456885 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:06.456827    2546 scope.go:117] "RemoveContainer" containerID="7e6428fedd73892d228bad589314745d674f1d5a5b37313a371cae55ed057b1f"
Jul 17 22:00:07.464568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:07.464476    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerID="dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0" exitCode=6
Jul 17 22:00:07.464568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:07.464535    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" event={"ID":"6dc1e0c9-bb05-442f-808f-2aa0314cdff0","Type":"ContainerDied","Data":"dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0"}
Jul 17 22:00:07.465081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:07.464590    2546 scope.go:117] "RemoveContainer" containerID="7e6428fedd73892d228bad589314745d674f1d5a5b37313a371cae55ed057b1f"
Jul 17 22:00:07.465081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:07.464755    2546 scope.go:117] "RemoveContainer" containerID="dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0"
Jul 17 22:00:07.465081 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:07.465008    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b_odh-ai-gateway-infra(6dc1e0c9-bb05-442f-808f-2aa0314cdff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0"
Jul 17 22:00:08.471120 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.471090    2546 scope.go:117] "RemoveContainer" containerID="dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0"
Jul 17 22:00:08.471504 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:08.471257    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b_odh-ai-gateway-infra(6dc1e0c9-bb05-442f-808f-2aa0314cdff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0"
Jul 17 22:00:08.791627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.791602    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 22:00:08.912103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912075    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912104    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912133    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912169    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912195    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912209    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912242    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912476    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:00:08.912568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912493    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:00:08.912568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912274    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69tmq\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912547    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:08.912568 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912548    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:00:08.912867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912596    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token\") pod \"781fb554-dea3-4b6e-a52d-0ebf80de2124\" (UID: \"781fb554-dea3-4b6e-a52d-0ebf80de2124\") "
Jul 17 22:00:08.912867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912832    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781fb554-dea3-4b6e-a52d-0ebf80de2124-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:08.912867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912856    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:08.913192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912871    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:08.913192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.912885    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:08.913304 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.913209    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data" (OuterVolumeSpecName: "istio-data") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:00:08.914372 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.914346    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:00:08.914583 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.914559    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:00:08.914657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.914619    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq" (OuterVolumeSpecName: "kube-api-access-69tmq") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "kube-api-access-69tmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:00:08.914832 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:08.914818    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token" (OuterVolumeSpecName: "istio-token") pod "781fb554-dea3-4b6e-a52d-0ebf80de2124" (UID: "781fb554-dea3-4b6e-a52d-0ebf80de2124"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:00:09.013972 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.013936    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.013972 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.013967    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.013972 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.013978    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.014137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.013986    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69tmq\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-kube-api-access-69tmq\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.014137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.013995    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781fb554-dea3-4b6e-a52d-0ebf80de2124-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:09.477607 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.477570    2546 generic.go:356] "Generic (PLEG): container finished" podID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerID="3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713" exitCode=0
Jul 17 22:00:09.478089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.477623    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"
Jul 17 22:00:09.478089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.477658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerDied","Data":"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"}
Jul 17 22:00:09.478089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.477743    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q" event={"ID":"781fb554-dea3-4b6e-a52d-0ebf80de2124","Type":"ContainerDied","Data":"200e192424594870584ab1d8d6d92e270b7d40c79cb8eb84ade74e34138d81d7"}
Jul 17 22:00:09.478089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.477780    2546 scope.go:117] "RemoveContainer" containerID="3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"
Jul 17 22:00:09.487167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.487151    2546 scope.go:117] "RemoveContainer" containerID="2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"
Jul 17 22:00:09.495224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.495208    2546 scope.go:117] "RemoveContainer" containerID="3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"
Jul 17 22:00:09.495469 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:09.495449    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713\": container with ID starting with 3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713 not found: ID does not exist" containerID="3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"
Jul 17 22:00:09.495523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.495479    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713"} err="failed to get container status \"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713\": rpc error: code = NotFound desc = could not find container \"3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713\": container with ID starting with 3c579cf14cb3489f42b64666486fc19d4d7a86dd09d7faf59c101ac7e9e6b713 not found: ID does not exist"
Jul 17 22:00:09.495523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.495495    2546 scope.go:117] "RemoveContainer" containerID="2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"
Jul 17 22:00:09.495775 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:09.495752    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5\": container with ID starting with 2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5 not found: ID does not exist" containerID="2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"
Jul 17 22:00:09.495821 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.495786    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5"} err="failed to get container status \"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5\": rpc error: code = NotFound desc = could not find container \"2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5\": container with ID starting with 2e2e34c2e9b1e93786283fb0814b0437d28a087e014c84a46de0b6319b9f93d5 not found: ID does not exist"
Jul 17 22:00:09.498900 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.498880    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 22:00:09.503010 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:09.502990    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-6e20ff19-openshift-default-5578479b55-vb54q"]
Jul 17 22:00:10.536220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:10.536185    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" path="/var/lib/kubelet/pods/781fb554-dea3-4b6e-a52d-0ebf80de2124/volumes"
Jul 17 22:00:21.531222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:21.531189    2546 scope.go:117] "RemoveContainer" containerID="dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0"
Jul 17 22:00:22.548459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:22.548422    2546 generic.go:356] "Generic (PLEG): container finished" podID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerID="c1163d38123ef48444f5ebca1971c7013c6ac128ba5182bd35d4d83d61374f87" exitCode=6
Jul 17 22:00:22.548901 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:22.548481    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" event={"ID":"6dc1e0c9-bb05-442f-808f-2aa0314cdff0","Type":"ContainerDied","Data":"c1163d38123ef48444f5ebca1971c7013c6ac128ba5182bd35d4d83d61374f87"}
Jul 17 22:00:22.548901 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:22.548546    2546 scope.go:117] "RemoveContainer" containerID="dbddf2cf6fc529a2a95e149438c4dd29ce9fb858b20ed1480bef22a9550289b0"
Jul 17 22:00:22.548901 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:22.548878    2546 scope.go:117] "RemoveContainer" containerID="c1163d38123ef48444f5ebca1971c7013c6ac128ba5182bd35d4d83d61374f87"
Jul 17 22:00:22.549109 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:22.549089    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b_odh-ai-gateway-infra(6dc1e0c9-bb05-442f-808f-2aa0314cdff0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0"
Jul 17 22:00:23.112962 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.112929    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:00:23.113358 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113340    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.113439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113361    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.113439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113386    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:23.113439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113395    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:23.113439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113405    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.113439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113414    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.113594 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113511    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="356116b1-208b-4c92-9d90-a03937e3213d" containerName="revoke-keys"
Jul 17 22:00:23.113594 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113526    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.113594 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.113537    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="781fb554-dea3-4b6e-a52d-0ebf80de2124" containerName="istio-proxy"
Jul 17 22:00:23.117776 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.117759    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.120216 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.120196    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-479788cb-openshift-default-dockercfg-rkqm8\""
Jul 17 22:00:23.129334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.129311    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:00:23.218492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218463    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218638 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218496    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218638 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218516    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218638 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218622    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218648    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218669    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218709    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218890 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218779    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xmz6\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.218890 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.218817    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.319907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.319874    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.319907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.319911    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.319933    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xmz6\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.319976    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320031    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320058    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320134    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320155    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320321    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320440    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320566 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320505    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320616 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320559    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.320877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.320859    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.322394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.322371    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.322592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.322576    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.331149 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.331127    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xmz6\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.331315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.331297    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token\") pod \"e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") " pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.426946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.426877    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:23.560881 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.560855    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:00:23.562514 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:23.562491    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8636e122_37c6_42e4_8464_76d227cb5c71.slice/crio-0ddaf285d6c10e606062c0910afa4323ae531b8955712833ce6dff8a7dc2c85a WatchSource:0}: Error finding container 0ddaf285d6c10e606062c0910afa4323ae531b8955712833ce6dff8a7dc2c85a: Status 404 returned error can't find the container with id 0ddaf285d6c10e606062c0910afa4323ae531b8955712833ce6dff8a7dc2c85a
Jul 17 22:00:23.564363 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.564333    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:23.564466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.564393    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:23.564466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.564421    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:23.574437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.574413    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:23.727113 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.727089    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:23.825075 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.825043    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca\") pod \"6dc1e0c9-bb05-442f-808f-2aa0314cdff0\" (UID: \"6dc1e0c9-bb05-442f-808f-2aa0314cdff0\") "
Jul 17 22:00:23.825424 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.825398    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6dc1e0c9-bb05-442f-808f-2aa0314cdff0" (UID: "6dc1e0c9-bb05-442f-808f-2aa0314cdff0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:23.925921 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:23.925892    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dc1e0c9-bb05-442f-808f-2aa0314cdff0-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:24.561352 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.561316    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b" event={"ID":"6dc1e0c9-bb05-442f-808f-2aa0314cdff0","Type":"ContainerDied","Data":"087540aac7edeae5d4718212d8cf895e1d5a28b330dd9eb0a60771a7debf82db"}
Jul 17 22:00:24.561763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.561384    2546 scope.go:117] "RemoveContainer" containerID="c1163d38123ef48444f5ebca1971c7013c6ac128ba5182bd35d4d83d61374f87"
Jul 17 22:00:24.561763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.561331    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"
Jul 17 22:00:24.563107 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.563081    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerStarted","Data":"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"}
Jul 17 22:00:24.563295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.563272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerStarted","Data":"0ddaf285d6c10e606062c0910afa4323ae531b8955712833ce6dff8a7dc2c85a"}
Jul 17 22:00:24.577349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.577325    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:24.581738 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.581676    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cmj5b"]
Jul 17 22:00:24.597876 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:24.597835    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podStartSLOduration=1.597823509 podStartE2EDuration="1.597823509s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:24.596884717 +0000 UTC m=+2600.577255203" watchObservedRunningTime="2026-07-17 22:00:24.597823509 +0000 UTC m=+2600.578193986"
Jul 17 22:00:25.427189 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:25.427158    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:25.431868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:25.431845    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:25.569125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:25.569096    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:25.570138 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:25.570116    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:26.535898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:26.535867    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" path="/var/lib/kubelet/pods/6dc1e0c9-bb05-442f-808f-2aa0314cdff0/volumes"
Jul 17 22:00:27.876160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876125    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876461    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876473    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876482    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876487    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876494    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876499    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876742 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876546    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.876742 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.876556    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:27.880754 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.880735    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:27.883276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.883253    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:00:27.887318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.887294    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:27.965227 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:27.965194    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r\" (UID: \"33a065e1-f3da-4289-b7ab-5c226ffff177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:28.066101 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.066067    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r\" (UID: \"33a065e1-f3da-4289-b7ab-5c226ffff177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:28.066665 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.066646    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r\" (UID: \"33a065e1-f3da-4289-b7ab-5c226ffff177\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:28.191840 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.191777    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:28.320885 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.320861    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:28.322934 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:28.322909    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33a065e1_f3da_4289_b7ab_5c226ffff177.slice/crio-01df59b04f1470b5ca7f0d5e10db9fd6b74cba0d96a9ad0dbe2a224bffb2e681 WatchSource:0}: Error finding container 01df59b04f1470b5ca7f0d5e10db9fd6b74cba0d96a9ad0dbe2a224bffb2e681: Status 404 returned error can't find the container with id 01df59b04f1470b5ca7f0d5e10db9fd6b74cba0d96a9ad0dbe2a224bffb2e681
Jul 17 22:00:28.584237 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.584211    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" event={"ID":"33a065e1-f3da-4289-b7ab-5c226ffff177","Type":"ContainerStarted","Data":"01df59b04f1470b5ca7f0d5e10db9fd6b74cba0d96a9ad0dbe2a224bffb2e681"}
Jul 17 22:00:28.818485 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.818457    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:28.818816 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.818804    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc1e0c9-bb05-442f-808f-2aa0314cdff0" containerName="revoke-keys"
Jul 17 22:00:28.821772 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.821756    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:28.829857 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.829832    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:28.973001 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:28.972974    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg\" (UID: \"eb62f9da-7fbc-4468-84c7-275c498240b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:29.074192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.074164    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg\" (UID: \"eb62f9da-7fbc-4468-84c7-275c498240b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:29.074732 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.074700    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg\" (UID: \"eb62f9da-7fbc-4468-84c7-275c498240b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:29.132238 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.132193    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:29.258691 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.258665    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:29.261015 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:29.260988    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb62f9da_7fbc_4468_84c7_275c498240b8.slice/crio-c670b83b2b7d99cc82541ecaa2f420ff0e3031edc30131b136713bdaec290e0a WatchSource:0}: Error finding container c670b83b2b7d99cc82541ecaa2f420ff0e3031edc30131b136713bdaec290e0a: Status 404 returned error can't find the container with id c670b83b2b7d99cc82541ecaa2f420ff0e3031edc30131b136713bdaec290e0a
Jul 17 22:00:29.590545 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.590515    2546 generic.go:356] "Generic (PLEG): container finished" podID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerID="2b3560a703f0fb36723bca3bbdb74eabbf7f9339f86bdcee35a1aee8cf46d1af" exitCode=6
Jul 17 22:00:29.590659 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.590581    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" event={"ID":"33a065e1-f3da-4289-b7ab-5c226ffff177","Type":"ContainerDied","Data":"2b3560a703f0fb36723bca3bbdb74eabbf7f9339f86bdcee35a1aee8cf46d1af"}
Jul 17 22:00:29.590825 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.590797    2546 scope.go:117] "RemoveContainer" containerID="2b3560a703f0fb36723bca3bbdb74eabbf7f9339f86bdcee35a1aee8cf46d1af"
Jul 17 22:00:29.592284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.592258    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerStarted","Data":"d0b69f4ef87bb31d4d880fe069e4de38cdfe7c1b91a7b9da195a4a216d420ee7"}
Jul 17 22:00:29.592381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.592308    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerStarted","Data":"c670b83b2b7d99cc82541ecaa2f420ff0e3031edc30131b136713bdaec290e0a"}
Jul 17 22:00:29.623570 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:29.623531    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" podStartSLOduration=1.62351874 podStartE2EDuration="1.62351874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:29.621226581 +0000 UTC m=+2605.601597061" watchObservedRunningTime="2026-07-17 22:00:29.62351874 +0000 UTC m=+2605.603889219"
Jul 17 22:00:30.598198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.598170    2546 generic.go:356] "Generic (PLEG): container finished" podID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerID="ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602" exitCode=6
Jul 17 22:00:30.598645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.598238    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" event={"ID":"33a065e1-f3da-4289-b7ab-5c226ffff177","Type":"ContainerDied","Data":"ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602"}
Jul 17 22:00:30.598645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.598286    2546 scope.go:117] "RemoveContainer" containerID="2b3560a703f0fb36723bca3bbdb74eabbf7f9339f86bdcee35a1aee8cf46d1af"
Jul 17 22:00:30.598645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.598433    2546 scope.go:117] "RemoveContainer" containerID="ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602"
Jul 17 22:00:30.598645 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:30.598621    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r_odh-ai-gateway-infra(33a065e1-f3da-4289-b7ab-5c226ffff177)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177"
Jul 17 22:00:30.599941 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.599921    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerID="d0b69f4ef87bb31d4d880fe069e4de38cdfe7c1b91a7b9da195a4a216d420ee7" exitCode=6
Jul 17 22:00:30.600059 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.599990    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerDied","Data":"d0b69f4ef87bb31d4d880fe069e4de38cdfe7c1b91a7b9da195a4a216d420ee7"}
Jul 17 22:00:30.600158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:30.600145    2546 scope.go:117] "RemoveContainer" containerID="d0b69f4ef87bb31d4d880fe069e4de38cdfe7c1b91a7b9da195a4a216d420ee7"
Jul 17 22:00:31.606498 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:31.606468    2546 scope.go:117] "RemoveContainer" containerID="ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602"
Jul 17 22:00:31.606937 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:31.606666    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r_odh-ai-gateway-infra(33a065e1-f3da-4289-b7ab-5c226ffff177)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177"
Jul 17 22:00:31.608048 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:31.608024    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerID="e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26" exitCode=6
Jul 17 22:00:31.608166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:31.608085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerDied","Data":"e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26"}
Jul 17 22:00:31.608166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:31.608137    2546 scope.go:117] "RemoveContainer" containerID="d0b69f4ef87bb31d4d880fe069e4de38cdfe7c1b91a7b9da195a4a216d420ee7"
Jul 17 22:00:31.608273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:31.608259    2546 scope.go:117] "RemoveContainer" containerID="e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26"
Jul 17 22:00:31.608436 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:31.608415    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg_odh-ai-gateway-infra(eb62f9da-7fbc-4468-84c7-275c498240b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8"
Jul 17 22:00:32.615100 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:32.615070    2546 scope.go:117] "RemoveContainer" containerID="e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26"
Jul 17 22:00:32.615446 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:32.615274    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg_odh-ai-gateway-infra(eb62f9da-7fbc-4468-84c7-275c498240b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8"
Jul 17 22:00:40.594955 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.594262    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:57768->10.133.0.100:15021: read: connection reset by peer" start-of-body=
Jul 17 22:00:40.594955 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.594331    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": read tcp 10.133.0.2:57768->10.133.0.100:15021: read: connection reset by peer"
Jul 17 22:00:40.652198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.652165    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:00:40.655522 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.655503    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:40.657836 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.657811    2546 generic.go:356] "Generic (PLEG): container finished" podID="8636e122-37c6-42e4-8464-76d227cb5c71" containerID="c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05" exitCode=0
Jul 17 22:00:40.657929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.657864    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerDied","Data":"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"}
Jul 17 22:00:40.658211 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.658193    2546 scope.go:117] "RemoveContainer" containerID="c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"
Jul 17 22:00:40.659301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.659274    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:40.659363 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.659335    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:40.659402 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.659365    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:00:40.664883 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.664862    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:00:40.671193 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.671167    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr\" (UID: \"6a5679c0-587e-4d44-b578-ba1d2bef0320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:40.772077 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.772017    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr\" (UID: \"6a5679c0-587e-4d44-b578-ba1d2bef0320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:40.772632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.772607    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr\" (UID: \"6a5679c0-587e-4d44-b578-ba1d2bef0320\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:40.968810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:40.968733    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:41.113258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:41.113230    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:00:41.115361 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:41.115328    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a5679c0_587e_4d44_b578_ba1d2bef0320.slice/crio-0e321cc1d083002fac84aa70ae64039323dbb633203fc1bd85c0d7b6dd7c1fc4 WatchSource:0}: Error finding container 0e321cc1d083002fac84aa70ae64039323dbb633203fc1bd85c0d7b6dd7c1fc4: Status 404 returned error can't find the container with id 0e321cc1d083002fac84aa70ae64039323dbb633203fc1bd85c0d7b6dd7c1fc4
Jul 17 22:00:41.668279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:41.668162    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerStarted","Data":"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"}
Jul 17 22:00:41.676040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:41.676001    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerStarted","Data":"3813819019d609e2146343828ed6485a0d2497e3a449d41d2be07bc9368cf1e5"}
Jul 17 22:00:41.676212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:41.676064    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerStarted","Data":"0e321cc1d083002fac84aa70ae64039323dbb633203fc1bd85c0d7b6dd7c1fc4"}
Jul 17 22:00:41.709412 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:41.709355    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" podStartSLOduration=1.709340796 podStartE2EDuration="1.709340796s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:00:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:00:41.707939031 +0000 UTC m=+2617.688309508" watchObservedRunningTime="2026-07-17 22:00:41.709340796 +0000 UTC m=+2617.689711273"
Jul 17 22:00:42.427840 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.427812    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:42.428110 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.428088    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Jul 17 22:00:42.428159 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.428137    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Jul 17 22:00:42.692569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.692388    2546 generic.go:356] "Generic (PLEG): container finished" podID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerID="3813819019d609e2146343828ed6485a0d2497e3a449d41d2be07bc9368cf1e5" exitCode=6
Jul 17 22:00:42.692569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.692522    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerDied","Data":"3813819019d609e2146343828ed6485a0d2497e3a449d41d2be07bc9368cf1e5"}
Jul 17 22:00:42.693177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:42.693160    2546 scope.go:117] "RemoveContainer" containerID="3813819019d609e2146343828ed6485a0d2497e3a449d41d2be07bc9368cf1e5"
Jul 17 22:00:43.427683 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.427648    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Jul 17 22:00:43.427884 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.427701    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Jul 17 22:00:43.699745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.699648    2546 generic.go:356] "Generic (PLEG): container finished" podID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerID="aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f" exitCode=6
Jul 17 22:00:43.699745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.699689    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerDied","Data":"aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f"}
Jul 17 22:00:43.700191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.699773    2546 scope.go:117] "RemoveContainer" containerID="3813819019d609e2146343828ed6485a0d2497e3a449d41d2be07bc9368cf1e5"
Jul 17 22:00:43.700191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:43.699921    2546 scope.go:117] "RemoveContainer" containerID="aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f"
Jul 17 22:00:43.700191 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:43.700132    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr_odh-ai-gateway-infra(6a5679c0-587e-4d44-b578-ba1d2bef0320)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320"
Jul 17 22:00:44.428194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:44.428163    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Jul 17 22:00:44.428376 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:44.428213    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Jul 17 22:00:44.533778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:44.533751    2546 scope.go:117] "RemoveContainer" containerID="ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602"
Jul 17 22:00:44.706571 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:44.706501    2546 scope.go:117] "RemoveContainer" containerID="aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f"
Jul 17 22:00:44.706949 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:44.706672    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr_odh-ai-gateway-infra(6a5679c0-587e-4d44-b578-ba1d2bef0320)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320"
Jul 17 22:00:45.427294 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.427259    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused" start-of-body=
Jul 17 22:00:45.427460 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.427312    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.100:15021/healthz/ready\": dial tcp 10.133.0.100:15021: connect: connection refused"
Jul 17 22:00:45.531782 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.531752    2546 scope.go:117] "RemoveContainer" containerID="e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26"
Jul 17 22:00:45.712840 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.712784    2546 generic.go:356] "Generic (PLEG): container finished" podID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerID="6ea626993b196d783652c28ae2cbeb7c2cdbefdfd5b692f15c3933d8f98d819c" exitCode=6
Jul 17 22:00:45.713177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.712857    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" event={"ID":"33a065e1-f3da-4289-b7ab-5c226ffff177","Type":"ContainerDied","Data":"6ea626993b196d783652c28ae2cbeb7c2cdbefdfd5b692f15c3933d8f98d819c"}
Jul 17 22:00:45.713177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.712920    2546 scope.go:117] "RemoveContainer" containerID="ce058fdd9c71dd4f2ab7da500bf776f1eaf51bed68ed1a44f1f8adf48e50b602"
Jul 17 22:00:45.713287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:45.713193    2546 scope.go:117] "RemoveContainer" containerID="6ea626993b196d783652c28ae2cbeb7c2cdbefdfd5b692f15c3933d8f98d819c"
Jul 17 22:00:45.713411 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:45.713391    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r_odh-ai-gateway-infra(33a065e1-f3da-4289-b7ab-5c226ffff177)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177"
Jul 17 22:00:46.430772 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.430685    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:46.431019 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.430987    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:46.431607 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.431585    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:00:46.720209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.720127    2546 generic.go:356] "Generic (PLEG): container finished" podID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerID="aa1d078372ab063be550137a99ed56d268659d6fc8740c800303339334106c8f" exitCode=6
Jul 17 22:00:46.720209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.720199    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerDied","Data":"aa1d078372ab063be550137a99ed56d268659d6fc8740c800303339334106c8f"}
Jul 17 22:00:46.720687 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.720259    2546 scope.go:117] "RemoveContainer" containerID="e9fdfda288b89aa46c2233ba988c8e6a8e2be27dc0c84b984aedd9b68bb5fd26"
Jul 17 22:00:46.720687 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.720634    2546 scope.go:117] "RemoveContainer" containerID="aa1d078372ab063be550137a99ed56d268659d6fc8740c800303339334106c8f"
Jul 17 22:00:46.720870 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:46.720844    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg_odh-ai-gateway-infra(eb62f9da-7fbc-4468-84c7-275c498240b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8"
Jul 17 22:00:46.741430 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.741403    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:46.869426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.869398    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:46.923942 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.923905    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca\") pod \"33a065e1-f3da-4289-b7ab-5c226ffff177\" (UID: \"33a065e1-f3da-4289-b7ab-5c226ffff177\") "
Jul 17 22:00:46.924245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:46.924224    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "33a065e1-f3da-4289-b7ab-5c226ffff177" (UID: "33a065e1-f3da-4289-b7ab-5c226ffff177"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:47.024575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.024550    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/33a065e1-f3da-4289-b7ab-5c226ffff177-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:47.726851 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.726822    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"
Jul 17 22:00:47.727325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.726821    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r" event={"ID":"33a065e1-f3da-4289-b7ab-5c226ffff177","Type":"ContainerDied","Data":"01df59b04f1470b5ca7f0d5e10db9fd6b74cba0d96a9ad0dbe2a224bffb2e681"}
Jul 17 22:00:47.727325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.726959    2546 scope.go:117] "RemoveContainer" containerID="6ea626993b196d783652c28ae2cbeb7c2cdbefdfd5b692f15c3933d8f98d819c"
Jul 17 22:00:47.747295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.747269    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:47.754203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.754177    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:47.760681 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.760663    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-tmp6r"]
Jul 17 22:00:47.878493 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.878468    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:47.932929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.932897    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca\") pod \"eb62f9da-7fbc-4468-84c7-275c498240b8\" (UID: \"eb62f9da-7fbc-4468-84c7-275c498240b8\") "
Jul 17 22:00:47.933266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:47.933242    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb62f9da-7fbc-4468-84c7-275c498240b8" (UID: "eb62f9da-7fbc-4468-84c7-275c498240b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:48.033504 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.033479    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb62f9da-7fbc-4468-84c7-275c498240b8-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:48.536493 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.536464    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" path="/var/lib/kubelet/pods/33a065e1-f3da-4289-b7ab-5c226ffff177/volumes"
Jul 17 22:00:48.735297 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.735268    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"
Jul 17 22:00:48.735762 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.735309    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg" event={"ID":"eb62f9da-7fbc-4468-84c7-275c498240b8","Type":"ContainerDied","Data":"c670b83b2b7d99cc82541ecaa2f420ff0e3031edc30131b136713bdaec290e0a"}
Jul 17 22:00:48.735762 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.735368    2546 scope.go:117] "RemoveContainer" containerID="aa1d078372ab063be550137a99ed56d268659d6fc8740c800303339334106c8f"
Jul 17 22:00:48.752550 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.752519    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:48.758156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:48.758135    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cv2rg"]
Jul 17 22:00:50.536580 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:50.536545    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" path="/var/lib/kubelet/pods/eb62f9da-7fbc-4468-84c7-275c498240b8/volumes"
Jul 17 22:00:55.407569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407538    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407845    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407856    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407866    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407872    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407878    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407883    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407890    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407895    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407903    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407909    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407918    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407922    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407974    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407981    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407989    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.408088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.407995    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:00:55.412114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.412098    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:00:55.418900 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.418877    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:00:55.486757 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.486708    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7j558\" (UID: \"2d5ec15c-1596-401e-8eb6-73e5bc1400ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:00:55.587683 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.587649    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7j558\" (UID: \"2d5ec15c-1596-401e-8eb6-73e5bc1400ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:00:55.588300 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.588280    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7j558\" (UID: \"2d5ec15c-1596-401e-8eb6-73e5bc1400ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:00:55.723907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.723822    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:00:55.854926 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:55.854901    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:00:55.856921 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:00:55.856889    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d5ec15c_1596_401e_8eb6_73e5bc1400ed.slice/crio-e5169336a625ebfe3a12deddf2872dd7c369f0947b886fb618ba451d0faffb52 WatchSource:0}: Error finding container e5169336a625ebfe3a12deddf2872dd7c369f0947b886fb618ba451d0faffb52: Status 404 returned error can't find the container with id e5169336a625ebfe3a12deddf2872dd7c369f0947b886fb618ba451d0faffb52
Jul 17 22:00:56.531689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.531664    2546 scope.go:117] "RemoveContainer" containerID="aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f"
Jul 17 22:00:56.779759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.779706    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerStarted","Data":"799f92e006f4906478e2ad16623841d336540ab118d3ade100727481c1023c30"}
Jul 17 22:00:56.781158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.781132    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerID="b86466d75538c01dce0a683454fd85119ea5577f3a8402e9394db90fe4c4386d" exitCode=6
Jul 17 22:00:56.781270 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.781222    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" event={"ID":"2d5ec15c-1596-401e-8eb6-73e5bc1400ed","Type":"ContainerDied","Data":"b86466d75538c01dce0a683454fd85119ea5577f3a8402e9394db90fe4c4386d"}
Jul 17 22:00:56.781329 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.781271    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" event={"ID":"2d5ec15c-1596-401e-8eb6-73e5bc1400ed","Type":"ContainerStarted","Data":"e5169336a625ebfe3a12deddf2872dd7c369f0947b886fb618ba451d0faffb52"}
Jul 17 22:00:56.781329 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:56.781324    2546 scope.go:117] "RemoveContainer" containerID="b86466d75538c01dce0a683454fd85119ea5577f3a8402e9394db90fe4c4386d"
Jul 17 22:00:57.787773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.787743    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerID="1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711" exitCode=6
Jul 17 22:00:57.788236 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.787819    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" event={"ID":"2d5ec15c-1596-401e-8eb6-73e5bc1400ed","Type":"ContainerDied","Data":"1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711"}
Jul 17 22:00:57.788236 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.787882    2546 scope.go:117] "RemoveContainer" containerID="b86466d75538c01dce0a683454fd85119ea5577f3a8402e9394db90fe4c4386d"
Jul 17 22:00:57.788236 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.788062    2546 scope.go:117] "RemoveContainer" containerID="1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711"
Jul 17 22:00:57.788377 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:57.788276    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7j558_odh-ai-gateway-infra(2d5ec15c-1596-401e-8eb6-73e5bc1400ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed"
Jul 17 22:00:57.789986 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.789964    2546 generic.go:356] "Generic (PLEG): container finished" podID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerID="799f92e006f4906478e2ad16623841d336540ab118d3ade100727481c1023c30" exitCode=6
Jul 17 22:00:57.790083 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.790000    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerDied","Data":"799f92e006f4906478e2ad16623841d336540ab118d3ade100727481c1023c30"}
Jul 17 22:00:57.790254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.790241    2546 scope.go:117] "RemoveContainer" containerID="799f92e006f4906478e2ad16623841d336540ab118d3ade100727481c1023c30"
Jul 17 22:00:57.790425 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:57.790406    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr_odh-ai-gateway-infra(6a5679c0-587e-4d44-b578-ba1d2bef0320)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320"
Jul 17 22:00:57.798957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.798938    2546 scope.go:117] "RemoveContainer" containerID="aee36cbc497137aae7c33b15828ed0536c95d2e42a3b9ac5f7f5bda06292191f"
Jul 17 22:00:57.805917 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:57.805894    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:00:58.798481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:58.798458    2546 scope.go:117] "RemoveContainer" containerID="1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711"
Jul 17 22:00:58.798794 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:00:58.798664    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7j558_odh-ai-gateway-infra(2d5ec15c-1596-401e-8eb6-73e5bc1400ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed"
Jul 17 22:00:58.927198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:58.927175    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:59.014014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.013982    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca\") pod \"6a5679c0-587e-4d44-b578-ba1d2bef0320\" (UID: \"6a5679c0-587e-4d44-b578-ba1d2bef0320\") "
Jul 17 22:00:59.014325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.014301    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a5679c0-587e-4d44-b578-ba1d2bef0320" (UID: "6a5679c0-587e-4d44-b578-ba1d2bef0320"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:00:59.115396 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.115306    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a5679c0-587e-4d44-b578-ba1d2bef0320-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:00:59.803764 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.803737    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"
Jul 17 22:00:59.804186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.803710    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr" event={"ID":"6a5679c0-587e-4d44-b578-ba1d2bef0320","Type":"ContainerDied","Data":"0e321cc1d083002fac84aa70ae64039323dbb633203fc1bd85c0d7b6dd7c1fc4"}
Jul 17 22:00:59.804186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.803864    2546 scope.go:117] "RemoveContainer" containerID="799f92e006f4906478e2ad16623841d336540ab118d3ade100727481c1023c30"
Jul 17 22:00:59.826956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.826932    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:00:59.832605 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:00:59.832583    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7l5hr"]
Jul 17 22:01:00.536506 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:00.536471    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" path="/var/lib/kubelet/pods/6a5679c0-587e-4d44-b578-ba1d2bef0320/volumes"
Jul 17 22:01:10.530913 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:10.530881    2546 scope.go:117] "RemoveContainer" containerID="1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711"
Jul 17 22:01:11.866102 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:11.866066    2546 generic.go:356] "Generic (PLEG): container finished" podID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerID="28e44ac0afc746666d4667873032f410d4f6eb9c233776dea23ad0596dd23146" exitCode=6
Jul 17 22:01:11.866587 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:11.866135    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" event={"ID":"2d5ec15c-1596-401e-8eb6-73e5bc1400ed","Type":"ContainerDied","Data":"28e44ac0afc746666d4667873032f410d4f6eb9c233776dea23ad0596dd23146"}
Jul 17 22:01:11.866587 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:11.866197    2546 scope.go:117] "RemoveContainer" containerID="1641c2eb3fe9608752b8faee73807e1b2711a122dece0dc9d88799819deb2711"
Jul 17 22:01:11.866587 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:11.866523    2546 scope.go:117] "RemoveContainer" containerID="28e44ac0afc746666d4667873032f410d4f6eb9c233776dea23ad0596dd23146"
Jul 17 22:01:11.866785 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:11.866755    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7j558_odh-ai-gateway-infra(2d5ec15c-1596-401e-8eb6-73e5bc1400ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed"
Jul 17 22:01:12.897432 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:12.895583    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:01:13.030094 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.030072    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:01:13.129328 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.129303    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca\") pod \"2d5ec15c-1596-401e-8eb6-73e5bc1400ed\" (UID: \"2d5ec15c-1596-401e-8eb6-73e5bc1400ed\") "
Jul 17 22:01:13.129677 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.129655    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d5ec15c-1596-401e-8eb6-73e5bc1400ed" (UID: "2d5ec15c-1596-401e-8eb6-73e5bc1400ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:13.230120 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.230037    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d5ec15c-1596-401e-8eb6-73e5bc1400ed-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:01:13.878207 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.878178    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"
Jul 17 22:01:13.878388 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.878174    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558" event={"ID":"2d5ec15c-1596-401e-8eb6-73e5bc1400ed","Type":"ContainerDied","Data":"e5169336a625ebfe3a12deddf2872dd7c369f0947b886fb618ba451d0faffb52"}
Jul 17 22:01:13.878388 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.878311    2546 scope.go:117] "RemoveContainer" containerID="28e44ac0afc746666d4667873032f410d4f6eb9c233776dea23ad0596dd23146"
Jul 17 22:01:13.899687 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.899662    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:01:13.905166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:13.905135    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7j558"]
Jul 17 22:01:14.536738 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:14.536693    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" path="/var/lib/kubelet/pods/2d5ec15c-1596-401e-8eb6-73e5bc1400ed/volumes"
Jul 17 22:01:17.954348 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954308    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954761    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954779    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954792    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954799    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954819    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954826    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954837    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.954845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954844    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954921    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954935    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb62f9da-7fbc-4468-84c7-275c498240b8" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954944    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954953    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954962    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="33a065e1-f3da-4289-b7ab-5c226ffff177" containerName="revoke-keys"
Jul 17 22:01:17.955274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.954974    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:17.959180 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.959159    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:17.961577 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.961557    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:01:17.965663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:17.965642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:18.067301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.067258    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp\" (UID: \"874d2c3c-9b92-4345-8105-30778b2744d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:18.168237 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.168185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp\" (UID: \"874d2c3c-9b92-4345-8105-30778b2744d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:18.168817 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.168800    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp\" (UID: \"874d2c3c-9b92-4345-8105-30778b2744d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:18.270892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.270866    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:18.606997 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.606965    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:18.609090 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:01:18.609067    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod874d2c3c_9b92_4345_8105_30778b2744d0.slice/crio-546a6783b6714cacfdef2206c6445ff574401fd000698166ca51cc8df4f4b187 WatchSource:0}: Error finding container 546a6783b6714cacfdef2206c6445ff574401fd000698166ca51cc8df4f4b187: Status 404 returned error can't find the container with id 546a6783b6714cacfdef2206c6445ff574401fd000698166ca51cc8df4f4b187
Jul 17 22:01:18.860651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.860572    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:18.861047 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861032    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:18.861181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861051    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:18.861181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861080    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:18.861181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861089    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:18.861181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861168    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a5679c0-587e-4d44-b578-ba1d2bef0320" containerName="revoke-keys"
Jul 17 22:01:18.861308 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.861182    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d5ec15c-1596-401e-8eb6-73e5bc1400ed" containerName="revoke-keys"
Jul 17 22:01:18.864421 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.864406    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:18.871411 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.871185    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:18.906289 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.906257    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerStarted","Data":"231a1a9fc908f9fd507cd03075a3f7788f05110a9f18ee4d36693583ee9f6e57"}
Jul 17 22:01:18.906417 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.906303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerStarted","Data":"546a6783b6714cacfdef2206c6445ff574401fd000698166ca51cc8df4f4b187"}
Jul 17 22:01:18.923844 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.923806    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" podStartSLOduration=1.923792774 podStartE2EDuration="1.923792774s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:18.921854302 +0000 UTC m=+2654.902224777" watchObservedRunningTime="2026-07-17 22:01:18.923792774 +0000 UTC m=+2654.904163250"
Jul 17 22:01:18.975032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:18.975007    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4\" (UID: \"77984794-0c96-41cb-9bff-1b32b0274396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:19.075817 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.075793    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4\" (UID: \"77984794-0c96-41cb-9bff-1b32b0274396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:19.076359 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.076343    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4\" (UID: \"77984794-0c96-41cb-9bff-1b32b0274396\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:19.175521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.175501    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:19.300658 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.300633    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:19.302363 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:01:19.302334    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77984794_0c96_41cb_9bff_1b32b0274396.slice/crio-722e4cd61846d344459056b8419758e824f64da3fbbd6f46139bb76a191b748d WatchSource:0}: Error finding container 722e4cd61846d344459056b8419758e824f64da3fbbd6f46139bb76a191b748d: Status 404 returned error can't find the container with id 722e4cd61846d344459056b8419758e824f64da3fbbd6f46139bb76a191b748d
Jul 17 22:01:19.319525 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.319500    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"]
Jul 17 22:01:19.324867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.324848    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:19.329779 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.329756    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"]
Jul 17 22:01:19.479652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.479626    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-479788cb-vszss\" (UID: \"285d910c-3ec2-4e95-9459-637ef0fa2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:19.580864 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.580836    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-479788cb-vszss\" (UID: \"285d910c-3ec2-4e95-9459-637ef0fa2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:19.581450 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.581426    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-479788cb-vszss\" (UID: \"285d910c-3ec2-4e95-9459-637ef0fa2144\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:19.645625 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.645595    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:19.775824 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.775798    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"]
Jul 17 22:01:19.777453 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:01:19.777402    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285d910c_3ec2_4e95_9459_637ef0fa2144.slice/crio-bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1 WatchSource:0}: Error finding container bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1: Status 404 returned error can't find the container with id bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1
Jul 17 22:01:19.912546 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.912510    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss" event={"ID":"285d910c-3ec2-4e95-9459-637ef0fa2144","Type":"ContainerStarted","Data":"bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1"}
Jul 17 22:01:19.913988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.913965    2546 generic.go:356] "Generic (PLEG): container finished" podID="77984794-0c96-41cb-9bff-1b32b0274396" containerID="b4b0d1e487d8d2e1d5dffc7e43f35ddfd94a1a5452d6c5f121d55f9fcf0d0899" exitCode=6
Jul 17 22:01:19.914124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.914058    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerDied","Data":"b4b0d1e487d8d2e1d5dffc7e43f35ddfd94a1a5452d6c5f121d55f9fcf0d0899"}
Jul 17 22:01:19.914124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.914108    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerStarted","Data":"722e4cd61846d344459056b8419758e824f64da3fbbd6f46139bb76a191b748d"}
Jul 17 22:01:19.914239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.914221    2546 scope.go:117] "RemoveContainer" containerID="b4b0d1e487d8d2e1d5dffc7e43f35ddfd94a1a5452d6c5f121d55f9fcf0d0899"
Jul 17 22:01:19.915657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.915633    2546 generic.go:356] "Generic (PLEG): container finished" podID="874d2c3c-9b92-4345-8105-30778b2744d0" containerID="231a1a9fc908f9fd507cd03075a3f7788f05110a9f18ee4d36693583ee9f6e57" exitCode=6
Jul 17 22:01:19.915783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.915743    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerDied","Data":"231a1a9fc908f9fd507cd03075a3f7788f05110a9f18ee4d36693583ee9f6e57"}
Jul 17 22:01:19.915853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:19.915841    2546 scope.go:117] "RemoveContainer" containerID="231a1a9fc908f9fd507cd03075a3f7788f05110a9f18ee4d36693583ee9f6e57"
Jul 17 22:01:20.923486 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.923452    2546 generic.go:356] "Generic (PLEG): container finished" podID="874d2c3c-9b92-4345-8105-30778b2744d0" containerID="1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd" exitCode=6
Jul 17 22:01:20.923946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.923538    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerDied","Data":"1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd"}
Jul 17 22:01:20.923946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.923617    2546 scope.go:117] "RemoveContainer" containerID="231a1a9fc908f9fd507cd03075a3f7788f05110a9f18ee4d36693583ee9f6e57"
Jul 17 22:01:20.923946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.923809    2546 scope.go:117] "RemoveContainer" containerID="1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd"
Jul 17 22:01:20.924088 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:20.924018    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp_odh-ai-gateway-infra(874d2c3c-9b92-4345-8105-30778b2744d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" podUID="874d2c3c-9b92-4345-8105-30778b2744d0"
Jul 17 22:01:20.925315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.925290    2546 generic.go:356] "Generic (PLEG): container finished" podID="285d910c-3ec2-4e95-9459-637ef0fa2144" containerID="2469176f3a3c077df86533cb76ea90a0fe8b8b167787eabdd3110862a4b8ac73" exitCode=0
Jul 17 22:01:20.925423 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.925358    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss" event={"ID":"285d910c-3ec2-4e95-9459-637ef0fa2144","Type":"ContainerDied","Data":"2469176f3a3c077df86533cb76ea90a0fe8b8b167787eabdd3110862a4b8ac73"}
Jul 17 22:01:20.927181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.927161    2546 generic.go:356] "Generic (PLEG): container finished" podID="77984794-0c96-41cb-9bff-1b32b0274396" containerID="9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082" exitCode=6
Jul 17 22:01:20.927273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.927208    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerDied","Data":"9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082"}
Jul 17 22:01:20.927391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.927377    2546 scope.go:117] "RemoveContainer" containerID="9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082"
Jul 17 22:01:20.927550 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:20.927535    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4_odh-ai-gateway-infra(77984794-0c96-41cb-9bff-1b32b0274396)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" podUID="77984794-0c96-41cb-9bff-1b32b0274396"
Jul 17 22:01:20.934968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:20.934947    2546 scope.go:117] "RemoveContainer" containerID="b4b0d1e487d8d2e1d5dffc7e43f35ddfd94a1a5452d6c5f121d55f9fcf0d0899"
Jul 17 22:01:21.934439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:21.934411    2546 scope.go:117] "RemoveContainer" containerID="9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082"
Jul 17 22:01:21.934933 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:21.934601    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4_odh-ai-gateway-infra(77984794-0c96-41cb-9bff-1b32b0274396)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" podUID="77984794-0c96-41cb-9bff-1b32b0274396"
Jul 17 22:01:21.936032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:21.936012    2546 scope.go:117] "RemoveContainer" containerID="1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd"
Jul 17 22:01:21.936370 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:21.936185    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp_odh-ai-gateway-infra(874d2c3c-9b92-4345-8105-30778b2744d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" podUID="874d2c3c-9b92-4345-8105-30778b2744d0"
Jul 17 22:01:22.066516 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.066495    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:22.203195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.203117    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca\") pod \"285d910c-3ec2-4e95-9459-637ef0fa2144\" (UID: \"285d910c-3ec2-4e95-9459-637ef0fa2144\") "
Jul 17 22:01:22.203433 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.203412    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "285d910c-3ec2-4e95-9459-637ef0fa2144" (UID: "285d910c-3ec2-4e95-9459-637ef0fa2144"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:22.304096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.304069    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/285d910c-3ec2-4e95-9459-637ef0fa2144-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:01:22.941763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.941669    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss"
Jul 17 22:01:22.941763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.941706    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-479788cb-vszss" event={"ID":"285d910c-3ec2-4e95-9459-637ef0fa2144","Type":"ContainerDied","Data":"bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1"}
Jul 17 22:01:22.941763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:22.941764    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bcf184f151a75ba3ccdd656a14b03d93ecf470b0be4696fe31bdc5041d3474f1"
Jul 17 22:01:30.689791 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.689753    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:30.690267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.690063    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="285d910c-3ec2-4e95-9459-637ef0fa2144" containerName="revoke-keys"
Jul 17 22:01:30.690267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.690074    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="285d910c-3ec2-4e95-9459-637ef0fa2144" containerName="revoke-keys"
Jul 17 22:01:30.690267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.690129    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="285d910c-3ec2-4e95-9459-637ef0fa2144" containerName="revoke-keys"
Jul 17 22:01:30.693275 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.693260    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:30.699905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.699880    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:30.876407 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.876373    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k\" (UID: \"5b0fe9c5-9491-469e-84f8-a192a03a806d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:30.977870 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.977799    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k\" (UID: \"5b0fe9c5-9491-469e-84f8-a192a03a806d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:30.978360 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:30.978339    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k\" (UID: \"5b0fe9c5-9491-469e-84f8-a192a03a806d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:31.005069 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.005044    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:31.135790 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.135770    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:31.138358 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:01:31.138326    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0fe9c5_9491_469e_84f8_a192a03a806d.slice/crio-c90ed27d5c2257e0683a0dfd6feeb982ffe3a17eae590f2186bc5ed7437ee8dd WatchSource:0}: Error finding container c90ed27d5c2257e0683a0dfd6feeb982ffe3a17eae590f2186bc5ed7437ee8dd: Status 404 returned error can't find the container with id c90ed27d5c2257e0683a0dfd6feeb982ffe3a17eae590f2186bc5ed7437ee8dd
Jul 17 22:01:31.993428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.993389    2546 generic.go:356] "Generic (PLEG): container finished" podID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerID="795e8bf509b72e9135a97993a1f94e1ce4b3d09072a22094f1c5aeee62bfefc4" exitCode=6
Jul 17 22:01:31.993863 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.993482    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerDied","Data":"795e8bf509b72e9135a97993a1f94e1ce4b3d09072a22094f1c5aeee62bfefc4"}
Jul 17 22:01:31.993863 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.993562    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerStarted","Data":"c90ed27d5c2257e0683a0dfd6feeb982ffe3a17eae590f2186bc5ed7437ee8dd"}
Jul 17 22:01:31.993863 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:31.993634    2546 scope.go:117] "RemoveContainer" containerID="795e8bf509b72e9135a97993a1f94e1ce4b3d09072a22094f1c5aeee62bfefc4"
Jul 17 22:01:33.000516 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:33.000483    2546 generic.go:356] "Generic (PLEG): container finished" podID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerID="2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c" exitCode=6
Jul 17 22:01:33.000923 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:33.000567    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerDied","Data":"2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c"}
Jul 17 22:01:33.000923 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:33.000632    2546 scope.go:117] "RemoveContainer" containerID="795e8bf509b72e9135a97993a1f94e1ce4b3d09072a22094f1c5aeee62bfefc4"
Jul 17 22:01:33.000923 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:33.000909    2546 scope.go:117] "RemoveContainer" containerID="2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c"
Jul 17 22:01:33.001119 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:33.001101    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k_odh-ai-gateway-infra(5b0fe9c5-9491-469e-84f8-a192a03a806d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d"
Jul 17 22:01:34.007164 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:34.007135    2546 scope.go:117] "RemoveContainer" containerID="2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c"
Jul 17 22:01:34.007634 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:34.007352    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k_odh-ai-gateway-infra(5b0fe9c5-9491-469e-84f8-a192a03a806d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d"
Jul 17 22:01:34.533650 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:34.533623    2546 scope.go:117] "RemoveContainer" containerID="1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd"
Jul 17 22:01:35.013350 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:35.013314    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerStarted","Data":"b3cc4942359b87d163a31f9dcfa9f45aadd21d66909cd2110b7e514706f0d4ee"}
Jul 17 22:01:35.531798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:35.531766    2546 scope.go:117] "RemoveContainer" containerID="9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082"
Jul 17 22:01:36.020563 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.020528    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerStarted","Data":"53fcd7d026731bcfc4563562fedd2ef96c9c7a962b60d48f48ff3caa569c525d"}
Jul 17 22:01:36.022418 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.022396    2546 generic.go:356] "Generic (PLEG): container finished" podID="874d2c3c-9b92-4345-8105-30778b2744d0" containerID="b3cc4942359b87d163a31f9dcfa9f45aadd21d66909cd2110b7e514706f0d4ee" exitCode=6
Jul 17 22:01:36.022532 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.022468    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerDied","Data":"b3cc4942359b87d163a31f9dcfa9f45aadd21d66909cd2110b7e514706f0d4ee"}
Jul 17 22:01:36.022532 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.022521    2546 scope.go:117] "RemoveContainer" containerID="1743ee9841661f87ec73cb92ca61199b532cdac52baf65e82217021ad58755bd"
Jul 17 22:01:36.022711 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.022697    2546 scope.go:117] "RemoveContainer" containerID="b3cc4942359b87d163a31f9dcfa9f45aadd21d66909cd2110b7e514706f0d4ee"
Jul 17 22:01:36.022949 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:36.022926    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp_odh-ai-gateway-infra(874d2c3c-9b92-4345-8105-30778b2744d0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" podUID="874d2c3c-9b92-4345-8105-30778b2744d0"
Jul 17 22:01:36.035649 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.035581    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" podStartSLOduration=18.035569706 podStartE2EDuration="18.035569706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:36.034469733 +0000 UTC m=+2672.014840213" watchObservedRunningTime="2026-07-17 22:01:36.035569706 +0000 UTC m=+2672.015940182"
Jul 17 22:01:36.039787 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:36.039763    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:37.029068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.029035    2546 generic.go:356] "Generic (PLEG): container finished" podID="77984794-0c96-41cb-9bff-1b32b0274396" containerID="53fcd7d026731bcfc4563562fedd2ef96c9c7a962b60d48f48ff3caa569c525d" exitCode=6
Jul 17 22:01:37.029549 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.029107    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerDied","Data":"53fcd7d026731bcfc4563562fedd2ef96c9c7a962b60d48f48ff3caa569c525d"}
Jul 17 22:01:37.029549 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.029175    2546 scope.go:117] "RemoveContainer" containerID="9b8354c8bd8b6d4fe952abbbcb6b842f99fa1da958a5960ed59942754cef4082"
Jul 17 22:01:37.029549 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.029486    2546 scope.go:117] "RemoveContainer" containerID="53fcd7d026731bcfc4563562fedd2ef96c9c7a962b60d48f48ff3caa569c525d"
Jul 17 22:01:37.029753 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:37.029711    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4_odh-ai-gateway-infra(77984794-0c96-41cb-9bff-1b32b0274396)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" podUID="77984794-0c96-41cb-9bff-1b32b0274396"
Jul 17 22:01:37.046158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.046133    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:37.159143 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.159121    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:37.229032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.229000    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca\") pod \"874d2c3c-9b92-4345-8105-30778b2744d0\" (UID: \"874d2c3c-9b92-4345-8105-30778b2744d0\") "
Jul 17 22:01:37.229343 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.229322    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "874d2c3c-9b92-4345-8105-30778b2744d0" (UID: "874d2c3c-9b92-4345-8105-30778b2744d0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:37.329608 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:37.329528    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/874d2c3c-9b92-4345-8105-30778b2744d0-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:01:38.037238 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.037204    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"
Jul 17 22:01:38.037745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.037240    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp" event={"ID":"874d2c3c-9b92-4345-8105-30778b2744d0","Type":"ContainerDied","Data":"546a6783b6714cacfdef2206c6445ff574401fd000698166ca51cc8df4f4b187"}
Jul 17 22:01:38.037745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.037339    2546 scope.go:117] "RemoveContainer" containerID="b3cc4942359b87d163a31f9dcfa9f45aadd21d66909cd2110b7e514706f0d4ee"
Jul 17 22:01:38.071258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.071227    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:38.075242 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.075213    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vllfp"]
Jul 17 22:01:38.170873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.170850    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:38.235003 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.234967    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca\") pod \"77984794-0c96-41cb-9bff-1b32b0274396\" (UID: \"77984794-0c96-41cb-9bff-1b32b0274396\") "
Jul 17 22:01:38.235350 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.235323    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "77984794-0c96-41cb-9bff-1b32b0274396" (UID: "77984794-0c96-41cb-9bff-1b32b0274396"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:38.335483 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.335402    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/77984794-0c96-41cb-9bff-1b32b0274396-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:01:38.536281 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:38.536251    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" path="/var/lib/kubelet/pods/874d2c3c-9b92-4345-8105-30778b2744d0/volumes"
Jul 17 22:01:39.045394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:39.045361    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4" event={"ID":"77984794-0c96-41cb-9bff-1b32b0274396","Type":"ContainerDied","Data":"722e4cd61846d344459056b8419758e824f64da3fbbd6f46139bb76a191b748d"}
Jul 17 22:01:39.045394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:39.045389    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"
Jul 17 22:01:39.045960 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:39.045416    2546 scope.go:117] "RemoveContainer" containerID="53fcd7d026731bcfc4563562fedd2ef96c9c7a962b60d48f48ff3caa569c525d"
Jul 17 22:01:39.062917 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:39.062892    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:39.065119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:39.065100    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-gtwd4"]
Jul 17 22:01:40.536801 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:40.536767    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77984794-0c96-41cb-9bff-1b32b0274396" path="/var/lib/kubelet/pods/77984794-0c96-41cb-9bff-1b32b0274396/volumes"
Jul 17 22:01:45.457543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457504    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457852    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457865    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457873    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457879    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457886    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457892    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457903    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457908    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457914    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457919    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457926    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.457968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457931    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.458375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.457992    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.458375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.458000    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:01:45.458375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.458008    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.458375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.458015    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.458375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.458021    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="77984794-0c96-41cb-9bff-1b32b0274396" containerName="revoke-keys"
Jul 17 22:01:45.461156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.461130    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:01:45.469537 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.469475    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:01:45.488240 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.488213    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27\" (UID: \"6d51b782-83e9-46ae-9fcb-48a05bf608db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:01:45.589625 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.589591    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27\" (UID: \"6d51b782-83e9-46ae-9fcb-48a05bf608db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:01:45.590192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.590172    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27\" (UID: \"6d51b782-83e9-46ae-9fcb-48a05bf608db\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:01:45.774448 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.774422    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:01:45.910886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:45.910856    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:01:45.912738 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:01:45.912698    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d51b782_83e9_46ae_9fcb_48a05bf608db.slice/crio-890107eace3964f86403422ed99053da715e50658ab2992245afd5a995adfb84 WatchSource:0}: Error finding container 890107eace3964f86403422ed99053da715e50658ab2992245afd5a995adfb84: Status 404 returned error can't find the container with id 890107eace3964f86403422ed99053da715e50658ab2992245afd5a995adfb84
Jul 17 22:01:46.086135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:46.086108    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" event={"ID":"6d51b782-83e9-46ae-9fcb-48a05bf608db","Type":"ContainerStarted","Data":"890107eace3964f86403422ed99053da715e50658ab2992245afd5a995adfb84"}
Jul 17 22:01:47.092284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:47.092251    2546 generic.go:356] "Generic (PLEG): container finished" podID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerID="7a9de2c748a4e79454f0937e378662e3bd92daaa680cfaade855f1a1a0b3602f" exitCode=6
Jul 17 22:01:47.092651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:47.092302    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" event={"ID":"6d51b782-83e9-46ae-9fcb-48a05bf608db","Type":"ContainerDied","Data":"7a9de2c748a4e79454f0937e378662e3bd92daaa680cfaade855f1a1a0b3602f"}
Jul 17 22:01:47.092651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:47.092479    2546 scope.go:117] "RemoveContainer" containerID="7a9de2c748a4e79454f0937e378662e3bd92daaa680cfaade855f1a1a0b3602f"
Jul 17 22:01:47.531058 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:47.531033    2546 scope.go:117] "RemoveContainer" containerID="2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c"
Jul 17 22:01:48.099420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.099329    2546 generic.go:356] "Generic (PLEG): container finished" podID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerID="06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0" exitCode=6
Jul 17 22:01:48.099420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.099410    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" event={"ID":"6d51b782-83e9-46ae-9fcb-48a05bf608db","Type":"ContainerDied","Data":"06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0"}
Jul 17 22:01:48.099950 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.099462    2546 scope.go:117] "RemoveContainer" containerID="7a9de2c748a4e79454f0937e378662e3bd92daaa680cfaade855f1a1a0b3602f"
Jul 17 22:01:48.099950 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.099637    2546 scope.go:117] "RemoveContainer" containerID="06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0"
Jul 17 22:01:48.099950 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:48.099898    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27_odh-ai-gateway-infra(6d51b782-83e9-46ae-9fcb-48a05bf608db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db"
Jul 17 22:01:48.101460 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.101442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerStarted","Data":"f0139d1bffe80d937dc888bb7832a51870db0777794a45e9a4524dd0722ad657"}
Jul 17 22:01:48.130977 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:48.130940    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" podStartSLOduration=18.130927902 podStartE2EDuration="18.130927902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:01:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:01:48.130338015 +0000 UTC m=+2684.110708493" watchObservedRunningTime="2026-07-17 22:01:48.130927902 +0000 UTC m=+2684.111298379"
Jul 17 22:01:49.108177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.108143    2546 generic.go:356] "Generic (PLEG): container finished" podID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerID="f0139d1bffe80d937dc888bb7832a51870db0777794a45e9a4524dd0722ad657" exitCode=6
Jul 17 22:01:49.108657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.108213    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerDied","Data":"f0139d1bffe80d937dc888bb7832a51870db0777794a45e9a4524dd0722ad657"}
Jul 17 22:01:49.108657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.108270    2546 scope.go:117] "RemoveContainer" containerID="2738cdf8b511f3390e9be98acefa831788cb968028736caeff3edc757a8c374c"
Jul 17 22:01:49.108657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.108551    2546 scope.go:117] "RemoveContainer" containerID="f0139d1bffe80d937dc888bb7832a51870db0777794a45e9a4524dd0722ad657"
Jul 17 22:01:49.108843 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:49.108788    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k_odh-ai-gateway-infra(5b0fe9c5-9491-469e-84f8-a192a03a806d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d"
Jul 17 22:01:49.110228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.110214    2546 scope.go:117] "RemoveContainer" containerID="06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0"
Jul 17 22:01:49.110390 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:01:49.110371    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27_odh-ai-gateway-infra(6d51b782-83e9-46ae-9fcb-48a05bf608db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db"
Jul 17 22:01:49.142184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:49.142147    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:50.251493 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:50.251471    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:50.324969 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:50.324946    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca\") pod \"5b0fe9c5-9491-469e-84f8-a192a03a806d\" (UID: \"5b0fe9c5-9491-469e-84f8-a192a03a806d\") "
Jul 17 22:01:50.325285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:50.325265    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5b0fe9c5-9491-469e-84f8-a192a03a806d" (UID: "5b0fe9c5-9491-469e-84f8-a192a03a806d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:01:50.426235 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:50.426176    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b0fe9c5-9491-469e-84f8-a192a03a806d-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:01:51.124861 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:51.124823    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k" event={"ID":"5b0fe9c5-9491-469e-84f8-a192a03a806d","Type":"ContainerDied","Data":"c90ed27d5c2257e0683a0dfd6feeb982ffe3a17eae590f2186bc5ed7437ee8dd"}
Jul 17 22:01:51.124861 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:51.124860    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"
Jul 17 22:01:51.125108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:51.124899    2546 scope.go:117] "RemoveContainer" containerID="f0139d1bffe80d937dc888bb7832a51870db0777794a45e9a4524dd0722ad657"
Jul 17 22:01:51.141026 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:51.141004    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:51.144822 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:51.144794    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-b9w5k"]
Jul 17 22:01:52.536021 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:01:52.535988    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" path="/var/lib/kubelet/pods/5b0fe9c5-9491-469e-84f8-a192a03a806d/volumes"
Jul 17 22:02:01.531363 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:01.531288    2546 scope.go:117] "RemoveContainer" containerID="06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0"
Jul 17 22:02:02.183929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:02.183902    2546 generic.go:356] "Generic (PLEG): container finished" podID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerID="b9f2091aaededda3e116b7b978fbd9d91675ac00f3739b0601e58f8dcc1466b6" exitCode=6
Jul 17 22:02:02.184123 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:02.183967    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" event={"ID":"6d51b782-83e9-46ae-9fcb-48a05bf608db","Type":"ContainerDied","Data":"b9f2091aaededda3e116b7b978fbd9d91675ac00f3739b0601e58f8dcc1466b6"}
Jul 17 22:02:02.184123 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:02.184033    2546 scope.go:117] "RemoveContainer" containerID="06b16fab35b7b486dadc988fe5c40b6d16c8dbba09848fb809e20bc926e820b0"
Jul 17 22:02:02.184370 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:02.184351    2546 scope.go:117] "RemoveContainer" containerID="b9f2091aaededda3e116b7b978fbd9d91675ac00f3739b0601e58f8dcc1466b6"
Jul 17 22:02:02.184579 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:02.184557    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27_odh-ai-gateway-infra(6d51b782-83e9-46ae-9fcb-48a05bf608db)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db"
Jul 17 22:02:03.226394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:03.226357    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:02:03.355671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:03.355647    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:02:03.423957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:03.423937    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca\") pod \"6d51b782-83e9-46ae-9fcb-48a05bf608db\" (UID: \"6d51b782-83e9-46ae-9fcb-48a05bf608db\") "
Jul 17 22:02:03.424280 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:03.424257    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d51b782-83e9-46ae-9fcb-48a05bf608db" (UID: "6d51b782-83e9-46ae-9fcb-48a05bf608db"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:03.525387 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:03.525359    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d51b782-83e9-46ae-9fcb-48a05bf608db-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:04.196617 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.196588    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"
Jul 17 22:02:04.196842 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.196629    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27" event={"ID":"6d51b782-83e9-46ae-9fcb-48a05bf608db","Type":"ContainerDied","Data":"890107eace3964f86403422ed99053da715e50658ab2992245afd5a995adfb84"}
Jul 17 22:02:04.196842 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.196700    2546 scope.go:117] "RemoveContainer" containerID="b9f2091aaededda3e116b7b978fbd9d91675ac00f3739b0601e58f8dcc1466b6"
Jul 17 22:02:04.218974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.218949    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:02:04.223060 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.223035    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hbq27"]
Jul 17 22:02:04.536557 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.536533    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" path="/var/lib/kubelet/pods/6d51b782-83e9-46ae-9fcb-48a05bf608db/volumes"
Jul 17 22:02:04.686689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.686664    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:02:04.696129 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:04.696109    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:02:17.988478 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988439    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988757    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988769    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988779    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988784    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988794    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988799    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988806    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988811    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988818    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.988865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988822    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.989171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988877    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.989171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988883    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:17.989171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988892    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.989171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988905    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:17.989171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.988912    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="874d2c3c-9b92-4345-8105-30778b2744d0" containerName="revoke-keys"
Jul 17 22:02:17.994953 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.992793    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:17.995743 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.995694    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:02:17.999515 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:17.999491    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:18.135406 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.135373    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw\" (UID: \"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:18.236667 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.236635    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw\" (UID: \"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:18.237264 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.237242    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw\" (UID: \"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:18.309053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.308977    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:18.438937 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.438917    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:18.441091 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:02:18.441063    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5c93a8c_7ba1_454d_a43d_5ce9e051a7f1.slice/crio-62c7ab450b3bd1312e923bc210dab89e05a3f4257c4565de5b5ae20e59512ed2 WatchSource:0}: Error finding container 62c7ab450b3bd1312e923bc210dab89e05a3f4257c4565de5b5ae20e59512ed2: Status 404 returned error can't find the container with id 62c7ab450b3bd1312e923bc210dab89e05a3f4257c4565de5b5ae20e59512ed2
Jul 17 22:02:18.911535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.911498    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:18.911870 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.911855    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:18.911918 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.911873    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:18.911957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.911942    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d51b782-83e9-46ae-9fcb-48a05bf608db" containerName="revoke-keys"
Jul 17 22:02:18.911957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.911953    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b0fe9c5-9491-469e-84f8-a192a03a806d" containerName="revoke-keys"
Jul 17 22:02:18.914944 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.914924    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:18.922974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:18.922920    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:19.042826 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.042793    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9\" (UID: \"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:19.143377 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.143350    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9\" (UID: \"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:19.143943 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.143926    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9\" (UID: \"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:19.225505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.225461    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:19.269098 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.269070    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerID="61062a8c91eb7f050770b274f889a4551cb05bd5c773a2ca26e73b23ce098455" exitCode=6
Jul 17 22:02:19.269245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.269158    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" event={"ID":"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1","Type":"ContainerDied","Data":"61062a8c91eb7f050770b274f889a4551cb05bd5c773a2ca26e73b23ce098455"}
Jul 17 22:02:19.269245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.269204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" event={"ID":"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1","Type":"ContainerStarted","Data":"62c7ab450b3bd1312e923bc210dab89e05a3f4257c4565de5b5ae20e59512ed2"}
Jul 17 22:02:19.269333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.269282    2546 scope.go:117] "RemoveContainer" containerID="61062a8c91eb7f050770b274f889a4551cb05bd5c773a2ca26e73b23ce098455"
Jul 17 22:02:19.355876 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:19.355856    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:19.357574 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:02:19.357547    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b7a2fb_5808_4008_9f83_9d3a53b55ef6.slice/crio-ee810544dbdabd50c0666202cfabf85095bb612e2ed11eb27567b893518cc4ef WatchSource:0}: Error finding container ee810544dbdabd50c0666202cfabf85095bb612e2ed11eb27567b893518cc4ef: Status 404 returned error can't find the container with id ee810544dbdabd50c0666202cfabf85095bb612e2ed11eb27567b893518cc4ef
Jul 17 22:02:20.276117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.276090    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerID="e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f" exitCode=6
Jul 17 22:02:20.276559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.276178    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" event={"ID":"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1","Type":"ContainerDied","Data":"e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f"}
Jul 17 22:02:20.276559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.276254    2546 scope.go:117] "RemoveContainer" containerID="61062a8c91eb7f050770b274f889a4551cb05bd5c773a2ca26e73b23ce098455"
Jul 17 22:02:20.276559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.276447    2546 scope.go:117] "RemoveContainer" containerID="e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f"
Jul 17 22:02:20.276735 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:20.276675    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw_odh-ai-gateway-infra(e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1"
Jul 17 22:02:20.277710 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.277689    2546 generic.go:356] "Generic (PLEG): container finished" podID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerID="4b0189de5d4ace6fae529a6da7aeafe5a0633e21d6cd7900fe801a590becaa35" exitCode=6
Jul 17 22:02:20.277818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.277753    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" event={"ID":"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6","Type":"ContainerDied","Data":"4b0189de5d4ace6fae529a6da7aeafe5a0633e21d6cd7900fe801a590becaa35"}
Jul 17 22:02:20.277818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.277796    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" event={"ID":"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6","Type":"ContainerStarted","Data":"ee810544dbdabd50c0666202cfabf85095bb612e2ed11eb27567b893518cc4ef"}
Jul 17 22:02:20.277965 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:20.277952    2546 scope.go:117] "RemoveContainer" containerID="4b0189de5d4ace6fae529a6da7aeafe5a0633e21d6cd7900fe801a590becaa35"
Jul 17 22:02:21.284715 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:21.284676    2546 generic.go:356] "Generic (PLEG): container finished" podID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerID="3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028" exitCode=6
Jul 17 22:02:21.285194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:21.284758    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" event={"ID":"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6","Type":"ContainerDied","Data":"3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028"}
Jul 17 22:02:21.285194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:21.284833    2546 scope.go:117] "RemoveContainer" containerID="4b0189de5d4ace6fae529a6da7aeafe5a0633e21d6cd7900fe801a590becaa35"
Jul 17 22:02:21.285194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:21.284986    2546 scope.go:117] "RemoveContainer" containerID="3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028"
Jul 17 22:02:21.285194 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:21.285184    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9_odh-ai-gateway-infra(a9b7a2fb-5808-4008-9f83-9d3a53b55ef6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6"
Jul 17 22:02:21.286842 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:21.286823    2546 scope.go:117] "RemoveContainer" containerID="e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f"
Jul 17 22:02:21.286991 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:21.286971    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw_odh-ai-gateway-infra(e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1"
Jul 17 22:02:22.294484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:22.294452    2546 scope.go:117] "RemoveContainer" containerID="3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028"
Jul 17 22:02:22.294967 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:22.294638    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9_odh-ai-gateway-infra(a9b7a2fb-5808-4008-9f83-9d3a53b55ef6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6"
Jul 17 22:02:26.518406 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:26.518364    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:02:26.518929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:26.518607    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" containerID="cri-o://92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c" gracePeriod=30
Jul 17 22:02:30.739652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.739616    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:30.744206 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.744186    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:30.750933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.750910    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:30.841490 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.841459    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht\" (UID: \"0727393f-c3c2-4c2b-a1e1-b52e668fa794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:30.942124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.942099    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht\" (UID: \"0727393f-c3c2-4c2b-a1e1-b52e668fa794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:30.942633 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:30.942615    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht\" (UID: \"0727393f-c3c2-4c2b-a1e1-b52e668fa794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:31.056162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.056102    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:31.187344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.187318    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:31.188934 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:02:31.188908    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0727393f_c3c2_4c2b_a1e1_b52e668fa794.slice/crio-ff3ca344dcf58de1ec6cb31895b765c4a48c398f490bc46a4a682d85a9a01d49 WatchSource:0}: Error finding container ff3ca344dcf58de1ec6cb31895b765c4a48c398f490bc46a4a682d85a9a01d49: Status 404 returned error can't find the container with id ff3ca344dcf58de1ec6cb31895b765c4a48c398f490bc46a4a682d85a9a01d49
Jul 17 22:02:31.338973 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.338896    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" event={"ID":"0727393f-c3c2-4c2b-a1e1-b52e668fa794","Type":"ContainerStarted","Data":"ff3ca344dcf58de1ec6cb31895b765c4a48c398f490bc46a4a682d85a9a01d49"}
Jul 17 22:02:31.432690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.432656    2546 patch_prober.go:28] interesting pod/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:02:31.432814 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.432714    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:02:31.752399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.752375    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:02:31.848983 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.848958    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.848990    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849007    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4xmz6\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849031    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849054    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849093    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849141    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849182    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849211    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket\") pod \"8636e122-37c6-42e4-8464-76d227cb5c71\" (UID: \"8636e122-37c6-42e4-8464-76d227cb5c71\") "
Jul 17 22:02:31.849540 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849348    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:31.849639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849598    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:02:31.849639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849627    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:02:31.849775 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.849529    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:02:31.850089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.850066    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data" (OuterVolumeSpecName: "istio-data") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:02:31.851344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.851312    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:02:31.851538 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.851520    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:02:31.851695 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.851681    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token" (OuterVolumeSpecName: "istio-token") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:02:31.851793 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.851695    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6" (OuterVolumeSpecName: "kube-api-access-4xmz6") pod "8636e122-37c6-42e4-8464-76d227cb5c71" (UID: "8636e122-37c6-42e4-8464-76d227cb5c71"). InnerVolumeSpecName "kube-api-access-4xmz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:02:31.950117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950056    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950080    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xmz6\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-kube-api-access-4xmz6\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950094    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8636e122-37c6-42e4-8464-76d227cb5c71-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950106    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950117 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950118    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950129    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950142    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8636e122-37c6-42e4-8464-76d227cb5c71-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950154    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8636e122-37c6-42e4-8464-76d227cb5c71-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:31.950340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:31.950165    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8636e122-37c6-42e4-8464-76d227cb5c71-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:32.345255 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.345222    2546 generic.go:356] "Generic (PLEG): container finished" podID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerID="613790f2824b35981d9065e3eea97c56106544622c90b82099989766fdd4788e" exitCode=6
Jul 17 22:02:32.345425 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.345303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" event={"ID":"0727393f-c3c2-4c2b-a1e1-b52e668fa794","Type":"ContainerDied","Data":"613790f2824b35981d9065e3eea97c56106544622c90b82099989766fdd4788e"}
Jul 17 22:02:32.345492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.345476    2546 scope.go:117] "RemoveContainer" containerID="613790f2824b35981d9065e3eea97c56106544622c90b82099989766fdd4788e"
Jul 17 22:02:32.347362 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.347336    2546 generic.go:356] "Generic (PLEG): container finished" podID="8636e122-37c6-42e4-8464-76d227cb5c71" containerID="92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c" exitCode=0
Jul 17 22:02:32.347473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.347383    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"
Jul 17 22:02:32.347473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.347389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerDied","Data":"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"}
Jul 17 22:02:32.347473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.347426    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp" event={"ID":"8636e122-37c6-42e4-8464-76d227cb5c71","Type":"ContainerDied","Data":"0ddaf285d6c10e606062c0910afa4323ae531b8955712833ce6dff8a7dc2c85a"}
Jul 17 22:02:32.347473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.347449    2546 scope.go:117] "RemoveContainer" containerID="92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"
Jul 17 22:02:32.356674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.356656    2546 scope.go:117] "RemoveContainer" containerID="c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"
Jul 17 22:02:32.370672 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.370654    2546 scope.go:117] "RemoveContainer" containerID="92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"
Jul 17 22:02:32.370936 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:32.370918    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c\": container with ID starting with 92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c not found: ID does not exist" containerID="92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"
Jul 17 22:02:32.371002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.370945    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c"} err="failed to get container status \"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c\": rpc error: code = NotFound desc = could not find container \"92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c\": container with ID starting with 92cbd65ddfb51b8729fcf2a229233da768dea1a11df888897e34cdc0e2b9902c not found: ID does not exist"
Jul 17 22:02:32.371002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.370962    2546 scope.go:117] "RemoveContainer" containerID="c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"
Jul 17 22:02:32.371229 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:32.371211    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05\": container with ID starting with c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05 not found: ID does not exist" containerID="c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"
Jul 17 22:02:32.371272 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.371235    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05"} err="failed to get container status \"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05\": rpc error: code = NotFound desc = could not find container \"c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05\": container with ID starting with c90aa659987b3cce7cec0c64cf63337e28b99040d92a90ca7249d51e70dbaf05 not found: ID does not exist"
Jul 17 22:02:32.375213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.375189    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:02:32.382918 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.382897    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-479788cb-openshift-default-5487b96d6c-2tpgp"]
Jul 17 22:02:32.536990 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:32.536963    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" path="/var/lib/kubelet/pods/8636e122-37c6-42e4-8464-76d227cb5c71/volumes"
Jul 17 22:02:33.355255 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:33.355223    2546 generic.go:356] "Generic (PLEG): container finished" podID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerID="619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3" exitCode=6
Jul 17 22:02:33.355690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:33.355270    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" event={"ID":"0727393f-c3c2-4c2b-a1e1-b52e668fa794","Type":"ContainerDied","Data":"619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3"}
Jul 17 22:02:33.355690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:33.355318    2546 scope.go:117] "RemoveContainer" containerID="613790f2824b35981d9065e3eea97c56106544622c90b82099989766fdd4788e"
Jul 17 22:02:33.355690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:33.355506    2546 scope.go:117] "RemoveContainer" containerID="619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3"
Jul 17 22:02:33.355873 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:33.355781    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht_odh-ai-gateway-infra(0727393f-c3c2-4c2b-a1e1-b52e668fa794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794"
Jul 17 22:02:33.531777 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:33.531743    2546 scope.go:117] "RemoveContainer" containerID="3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028"
Jul 17 22:02:34.364818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.364251    2546 generic.go:356] "Generic (PLEG): container finished" podID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerID="0490f4880de7037b243757afd5deaf56d4d3e675e28a850a4681cab7c7f0be88" exitCode=6
Jul 17 22:02:34.364818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.364409    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" event={"ID":"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6","Type":"ContainerDied","Data":"0490f4880de7037b243757afd5deaf56d4d3e675e28a850a4681cab7c7f0be88"}
Jul 17 22:02:34.364818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.364463    2546 scope.go:117] "RemoveContainer" containerID="3883d5c40cbec617a793d60f68b61d5fff8acad01494fc1c81bc13e9da38a028"
Jul 17 22:02:34.364818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.364799    2546 scope.go:117] "RemoveContainer" containerID="0490f4880de7037b243757afd5deaf56d4d3e675e28a850a4681cab7c7f0be88"
Jul 17 22:02:34.365384 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:34.365020    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9_odh-ai-gateway-infra(a9b7a2fb-5808-4008-9f83-9d3a53b55ef6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6"
Jul 17 22:02:34.366954 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.366934    2546 scope.go:117] "RemoveContainer" containerID="619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3"
Jul 17 22:02:34.367089 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:34.367070    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht_odh-ai-gateway-infra(0727393f-c3c2-4c2b-a1e1-b52e668fa794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794"
Jul 17 22:02:34.534103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:34.534033    2546 scope.go:117] "RemoveContainer" containerID="e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f"
Jul 17 22:02:35.373907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.373877    2546 generic.go:356] "Generic (PLEG): container finished" podID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerID="2bd365aa497cd65d07616bab1b7df18a8a23b74b537c4b90471abd695bacb83e" exitCode=6
Jul 17 22:02:35.374292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.373950    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" event={"ID":"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1","Type":"ContainerDied","Data":"2bd365aa497cd65d07616bab1b7df18a8a23b74b537c4b90471abd695bacb83e"}
Jul 17 22:02:35.374292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.374016    2546 scope.go:117] "RemoveContainer" containerID="e3d8d4853aa9d357116e6d0bc57cab28adfcf165a4974af6df5a289dd413897f"
Jul 17 22:02:35.374412 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.374327    2546 scope.go:117] "RemoveContainer" containerID="2bd365aa497cd65d07616bab1b7df18a8a23b74b537c4b90471abd695bacb83e"
Jul 17 22:02:35.374556 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:35.374531    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw_odh-ai-gateway-infra(e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1"
Jul 17 22:02:35.390590 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.390564    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:35.501898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.501873    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:35.502180 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.502169    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:02:35.502225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.502182    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:02:35.502260 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.502252    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:02:35.506356 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.506338    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:35.513007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.512981    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:35.525281 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.525258    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:35.682697 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.682666    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca\") pod \"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6\" (UID: \"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6\") "
Jul 17 22:02:35.682895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.682795    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9\" (UID: \"51548ed5-80c0-43af-a05e-7361ad819829\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:35.683031 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.683011    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" (UID: "a9b7a2fb-5808-4008-9f83-9d3a53b55ef6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:35.783480 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.783449    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9\" (UID: \"51548ed5-80c0-43af-a05e-7361ad819829\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:35.783624 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.783524    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:35.784065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.784046    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9\" (UID: \"51548ed5-80c0-43af-a05e-7361ad819829\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:35.830553 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.830528    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:35.980041 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:35.980017    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:35.981661 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:02:35.981631    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51548ed5_80c0_43af_a05e_7361ad819829.slice/crio-1d29f71d89c17cef8165e937afa2acffdca13c5fa0c1f19b05909ca43a94dac9 WatchSource:0}: Error finding container 1d29f71d89c17cef8165e937afa2acffdca13c5fa0c1f19b05909ca43a94dac9: Status 404 returned error can't find the container with id 1d29f71d89c17cef8165e937afa2acffdca13c5fa0c1f19b05909ca43a94dac9
Jul 17 22:02:36.382659 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.382631    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerStarted","Data":"695f0ecbb9127c24fb92f394992c453fa84b272aaa09ef6169d096d4939f9b3d"}
Jul 17 22:02:36.383080 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.382679    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerStarted","Data":"1d29f71d89c17cef8165e937afa2acffdca13c5fa0c1f19b05909ca43a94dac9"}
Jul 17 22:02:36.385245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.385217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9" event={"ID":"a9b7a2fb-5808-4008-9f83-9d3a53b55ef6","Type":"ContainerDied","Data":"ee810544dbdabd50c0666202cfabf85095bb612e2ed11eb27567b893518cc4ef"}
Jul 17 22:02:36.385245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.385230    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"
Jul 17 22:02:36.385397 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.385261    2546 scope.go:117] "RemoveContainer" containerID="0490f4880de7037b243757afd5deaf56d4d3e675e28a850a4681cab7c7f0be88"
Jul 17 22:02:36.401527 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.401472    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" podStartSLOduration=1.401454712 podStartE2EDuration="1.401454712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:02:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:02:36.401293843 +0000 UTC m=+2732.381664320" watchObservedRunningTime="2026-07-17 22:02:36.401454712 +0000 UTC m=+2732.381825189"
Jul 17 22:02:36.407536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.407515    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:36.423899 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.423845    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:36.430681 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.430658    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-dk2l9"]
Jul 17 22:02:36.537261 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.537234    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" path="/var/lib/kubelet/pods/a9b7a2fb-5808-4008-9f83-9d3a53b55ef6/volumes"
Jul 17 22:02:36.538216 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.538201    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:36.691355 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.691295    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca\") pod \"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1\" (UID: \"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1\") "
Jul 17 22:02:36.691657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.691636    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" (UID: "e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:36.792809 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:36.792772    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:37.391395 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.391355    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw" event={"ID":"e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1","Type":"ContainerDied","Data":"62c7ab450b3bd1312e923bc210dab89e05a3f4257c4565de5b5ae20e59512ed2"}
Jul 17 22:02:37.391395 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.391365    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"
Jul 17 22:02:37.391953 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.391425    2546 scope.go:117] "RemoveContainer" containerID="2bd365aa497cd65d07616bab1b7df18a8a23b74b537c4b90471abd695bacb83e"
Jul 17 22:02:37.394472 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.394447    2546 generic.go:356] "Generic (PLEG): container finished" podID="51548ed5-80c0-43af-a05e-7361ad819829" containerID="695f0ecbb9127c24fb92f394992c453fa84b272aaa09ef6169d096d4939f9b3d" exitCode=6
Jul 17 22:02:37.394569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.394518    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerDied","Data":"695f0ecbb9127c24fb92f394992c453fa84b272aaa09ef6169d096d4939f9b3d"}
Jul 17 22:02:37.394792 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.394752    2546 scope.go:117] "RemoveContainer" containerID="695f0ecbb9127c24fb92f394992c453fa84b272aaa09ef6169d096d4939f9b3d"
Jul 17 22:02:37.431925 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.431899    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:37.432164 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:37.432145    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-l6cvw"]
Jul 17 22:02:38.402789 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:38.402756    2546 generic.go:356] "Generic (PLEG): container finished" podID="51548ed5-80c0-43af-a05e-7361ad819829" containerID="54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe" exitCode=6
Jul 17 22:02:38.403257 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:38.402836    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerDied","Data":"54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe"}
Jul 17 22:02:38.403257 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:38.402881    2546 scope.go:117] "RemoveContainer" containerID="695f0ecbb9127c24fb92f394992c453fa84b272aaa09ef6169d096d4939f9b3d"
Jul 17 22:02:38.403257 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:38.403144    2546 scope.go:117] "RemoveContainer" containerID="54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe"
Jul 17 22:02:38.403422 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:38.403363    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9_odh-ai-gateway-infra(51548ed5-80c0-43af-a05e-7361ad819829)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" podUID="51548ed5-80c0-43af-a05e-7361ad819829"
Jul 17 22:02:38.535827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:38.535793    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" path="/var/lib/kubelet/pods/e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1/volumes"
Jul 17 22:02:39.409340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:39.409314    2546 scope.go:117] "RemoveContainer" containerID="54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe"
Jul 17 22:02:39.409830 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:39.409545    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9_odh-ai-gateway-infra(51548ed5-80c0-43af-a05e-7361ad819829)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" podUID="51548ed5-80c0-43af-a05e-7361ad819829"
Jul 17 22:02:45.530800 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:45.530771    2546 scope.go:117] "RemoveContainer" containerID="619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3"
Jul 17 22:02:46.446292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:46.446209    2546 generic.go:356] "Generic (PLEG): container finished" podID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerID="2bd4b35245500731c775ad5219e9c10828e57dc5c194f037841309f103afa795" exitCode=6
Jul 17 22:02:46.446447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:46.446296    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" event={"ID":"0727393f-c3c2-4c2b-a1e1-b52e668fa794","Type":"ContainerDied","Data":"2bd4b35245500731c775ad5219e9c10828e57dc5c194f037841309f103afa795"}
Jul 17 22:02:46.446447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:46.446356    2546 scope.go:117] "RemoveContainer" containerID="619721a020f6f2040c2ad15f83db6d7432000d1106e8601bbf2fc5a05e0dc0f3"
Jul 17 22:02:46.446738 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:46.446705    2546 scope.go:117] "RemoveContainer" containerID="2bd4b35245500731c775ad5219e9c10828e57dc5c194f037841309f103afa795"
Jul 17 22:02:46.446946 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:46.446923    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht_odh-ai-gateway-infra(0727393f-c3c2-4c2b-a1e1-b52e668fa794)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794"
Jul 17 22:02:47.475607 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:47.475566    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:47.607835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:47.607809    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:47.666546 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:47.666519    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca\") pod \"0727393f-c3c2-4c2b-a1e1-b52e668fa794\" (UID: \"0727393f-c3c2-4c2b-a1e1-b52e668fa794\") "
Jul 17 22:02:47.666874 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:47.666851    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0727393f-c3c2-4c2b-a1e1-b52e668fa794" (UID: "0727393f-c3c2-4c2b-a1e1-b52e668fa794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:47.767686 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:47.767657    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0727393f-c3c2-4c2b-a1e1-b52e668fa794-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:48.458896 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.458870    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"
Jul 17 22:02:48.459053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.458914    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht" event={"ID":"0727393f-c3c2-4c2b-a1e1-b52e668fa794","Type":"ContainerDied","Data":"ff3ca344dcf58de1ec6cb31895b765c4a48c398f490bc46a4a682d85a9a01d49"}
Jul 17 22:02:48.459053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.458973    2546 scope.go:117] "RemoveContainer" containerID="2bd4b35245500731c775ad5219e9c10828e57dc5c194f037841309f103afa795"
Jul 17 22:02:48.481459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.481430    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:48.487453 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.487432    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-mwkht"]
Jul 17 22:02:48.536431 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:48.536406    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" path="/var/lib/kubelet/pods/0727393f-c3c2-4c2b-a1e1-b52e668fa794/volumes"
Jul 17 22:02:51.530979 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:51.530955    2546 scope.go:117] "RemoveContainer" containerID="54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe"
Jul 17 22:02:52.489612 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:52.489580    2546 generic.go:356] "Generic (PLEG): container finished" podID="51548ed5-80c0-43af-a05e-7361ad819829" containerID="12c7602505f97ff254f49526e544d4e1c06ea049136b14c082efa19fce3303a5" exitCode=6
Jul 17 22:02:52.489782 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:52.489655    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerDied","Data":"12c7602505f97ff254f49526e544d4e1c06ea049136b14c082efa19fce3303a5"}
Jul 17 22:02:52.489782 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:52.489715    2546 scope.go:117] "RemoveContainer" containerID="54f63ff0ee927f84e82ae458edf21f888b15978a323a5c27db3676437b50c3fe"
Jul 17 22:02:52.490035 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:52.490004    2546 scope.go:117] "RemoveContainer" containerID="12c7602505f97ff254f49526e544d4e1c06ea049136b14c082efa19fce3303a5"
Jul 17 22:02:52.490205 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:02:52.490187    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9_odh-ai-gateway-infra(51548ed5-80c0-43af-a05e-7361ad819829)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" podUID="51548ed5-80c0-43af-a05e-7361ad819829"
Jul 17 22:02:53.515209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:53.515169    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:53.648800 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:53.648778    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:53.708287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:53.708265    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca\") pod \"51548ed5-80c0-43af-a05e-7361ad819829\" (UID: \"51548ed5-80c0-43af-a05e-7361ad819829\") "
Jul 17 22:02:53.708582 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:53.708561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "51548ed5-80c0-43af-a05e-7361ad819829" (UID: "51548ed5-80c0-43af-a05e-7361ad819829"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:02:53.808805 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:53.808752    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/51548ed5-80c0-43af-a05e-7361ad819829-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:02:54.503713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:54.503686    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"
Jul 17 22:02:54.503888 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:54.503751    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9" event={"ID":"51548ed5-80c0-43af-a05e-7361ad819829","Type":"ContainerDied","Data":"1d29f71d89c17cef8165e937afa2acffdca13c5fa0c1f19b05909ca43a94dac9"}
Jul 17 22:02:54.503888 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:54.503808    2546 scope.go:117] "RemoveContainer" containerID="12c7602505f97ff254f49526e544d4e1c06ea049136b14c082efa19fce3303a5"
Jul 17 22:02:54.542040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:54.542016    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:54.542040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:54.542046    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-d7sp9"]
Jul 17 22:02:56.536438 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:02:56.536399    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="51548ed5-80c0-43af-a05e-7361ad819829" path="/var/lib/kubelet/pods/51548ed5-80c0-43af-a05e-7361ad819829/volumes"
Jul 17 22:03:08.032383 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032347    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032676    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032691    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032708    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032735    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032747    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032756    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032766    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032772    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032780    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032787    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032795    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032803    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032812    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032818    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032828    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.032833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032835    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032843    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032852    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032869    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032876    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032885    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032890    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032895    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032901    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032907    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032912    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032974    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032980    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032986    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032992    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8636e122-37c6-42e4-8464-76d227cb5c71" containerName="istio-proxy"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.032997    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033002    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033006    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033012    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033018    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0727393f-c3c2-4c2b-a1e1-b52e668fa794" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033023    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033027    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5c93a8c-7ba1-454d-a43d-5ce9e051a7f1" containerName="revoke-keys"
Jul 17 22:03:08.033325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.033031    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="51548ed5-80c0-43af-a05e-7361ad819829" containerName="revoke-keys"
Jul 17 22:03:08.037593 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.037576    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:08.040124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.040101    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:03:08.042711 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.042683    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:08.110276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.110243    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6\" (UID: \"c3bbf334-b2b3-432d-8792-4d052991c991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:08.210645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.210615    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6\" (UID: \"c3bbf334-b2b3-432d-8792-4d052991c991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:08.211217 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.211199    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6\" (UID: \"c3bbf334-b2b3-432d-8792-4d052991c991\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:08.348497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.348436    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:08.476922 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.476898    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:08.479336 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:08.479306    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3bbf334_b2b3_432d_8792_4d052991c991.slice/crio-c6666f043c8551c1cd2dd6e94ca33e2d502063c4eddf494f9720180ca99d33f3 WatchSource:0}: Error finding container c6666f043c8551c1cd2dd6e94ca33e2d502063c4eddf494f9720180ca99d33f3: Status 404 returned error can't find the container with id c6666f043c8551c1cd2dd6e94ca33e2d502063c4eddf494f9720180ca99d33f3
Jul 17 22:03:08.573778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.573741    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" event={"ID":"c3bbf334-b2b3-432d-8792-4d052991c991","Type":"ContainerStarted","Data":"c6666f043c8551c1cd2dd6e94ca33e2d502063c4eddf494f9720180ca99d33f3"}
Jul 17 22:03:08.952501 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.952422    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:08.952800 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.952787    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a9b7a2fb-5808-4008-9f83-9d3a53b55ef6" containerName="revoke-keys"
Jul 17 22:03:08.955659 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.955643    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:08.962664 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:08.962529    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:09.016858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.016825    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl\" (UID: \"1bbc2a2e-9c62-4f82-9dae-5e212630af1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:09.118092 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.118061    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl\" (UID: \"1bbc2a2e-9c62-4f82-9dae-5e212630af1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:09.118607 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.118588    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl\" (UID: \"1bbc2a2e-9c62-4f82-9dae-5e212630af1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:09.267741 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.267684    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:09.395643 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.395606    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:09.399509 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:09.399482    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bbc2a2e_9c62_4f82_9dae_5e212630af1c.slice/crio-841a535228f7e0fdc645c0a3b2823d90d395b30c99a672255dd0343de2a94143 WatchSource:0}: Error finding container 841a535228f7e0fdc645c0a3b2823d90d395b30c99a672255dd0343de2a94143: Status 404 returned error can't find the container with id 841a535228f7e0fdc645c0a3b2823d90d395b30c99a672255dd0343de2a94143
Jul 17 22:03:09.581300 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.581272    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3bbf334-b2b3-432d-8792-4d052991c991" containerID="8c9b0af64ca8e4665c4a39975a58a2ee3f815822f078dc78c2ab62d71b37240a" exitCode=6
Jul 17 22:03:09.581442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.581366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" event={"ID":"c3bbf334-b2b3-432d-8792-4d052991c991","Type":"ContainerDied","Data":"8c9b0af64ca8e4665c4a39975a58a2ee3f815822f078dc78c2ab62d71b37240a"}
Jul 17 22:03:09.581562 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.581546    2546 scope.go:117] "RemoveContainer" containerID="8c9b0af64ca8e4665c4a39975a58a2ee3f815822f078dc78c2ab62d71b37240a"
Jul 17 22:03:09.582635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:09.582614    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" event={"ID":"1bbc2a2e-9c62-4f82-9dae-5e212630af1c","Type":"ContainerStarted","Data":"841a535228f7e0fdc645c0a3b2823d90d395b30c99a672255dd0343de2a94143"}
Jul 17 22:03:10.589380 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.589349    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3bbf334-b2b3-432d-8792-4d052991c991" containerID="b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb" exitCode=6
Jul 17 22:03:10.589846 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.589445    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" event={"ID":"c3bbf334-b2b3-432d-8792-4d052991c991","Type":"ContainerDied","Data":"b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb"}
Jul 17 22:03:10.589846 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.589504    2546 scope.go:117] "RemoveContainer" containerID="8c9b0af64ca8e4665c4a39975a58a2ee3f815822f078dc78c2ab62d71b37240a"
Jul 17 22:03:10.589846 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.589612    2546 scope.go:117] "RemoveContainer" containerID="b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb"
Jul 17 22:03:10.590004 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:10.589846    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6_odh-ai-gateway-infra(c3bbf334-b2b3-432d-8792-4d052991c991)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" podUID="c3bbf334-b2b3-432d-8792-4d052991c991"
Jul 17 22:03:10.591075 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.591054    2546 generic.go:356] "Generic (PLEG): container finished" podID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerID="0d02f2048ed7f3529247d9871bb42f5d0c627706e1d30c1479495a313ca1eeea" exitCode=6
Jul 17 22:03:10.591158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.591100    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" event={"ID":"1bbc2a2e-9c62-4f82-9dae-5e212630af1c","Type":"ContainerDied","Data":"0d02f2048ed7f3529247d9871bb42f5d0c627706e1d30c1479495a313ca1eeea"}
Jul 17 22:03:10.591263 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:10.591252    2546 scope.go:117] "RemoveContainer" containerID="0d02f2048ed7f3529247d9871bb42f5d0c627706e1d30c1479495a313ca1eeea"
Jul 17 22:03:11.597833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:11.597804    2546 scope.go:117] "RemoveContainer" containerID="b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb"
Jul 17 22:03:11.598229 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:11.597984    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6_odh-ai-gateway-infra(c3bbf334-b2b3-432d-8792-4d052991c991)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" podUID="c3bbf334-b2b3-432d-8792-4d052991c991"
Jul 17 22:03:11.599428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:11.599408    2546 generic.go:356] "Generic (PLEG): container finished" podID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerID="39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0" exitCode=6
Jul 17 22:03:11.599501 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:11.599456    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" event={"ID":"1bbc2a2e-9c62-4f82-9dae-5e212630af1c","Type":"ContainerDied","Data":"39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0"}
Jul 17 22:03:11.599501 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:11.599495    2546 scope.go:117] "RemoveContainer" containerID="0d02f2048ed7f3529247d9871bb42f5d0c627706e1d30c1479495a313ca1eeea"
Jul 17 22:03:11.599630 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:11.599615    2546 scope.go:117] "RemoveContainer" containerID="39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0"
Jul 17 22:03:11.599828 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:11.599805    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl_odh-ai-gateway-infra(1bbc2a2e-9c62-4f82-9dae-5e212630af1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c"
Jul 17 22:03:12.605645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:12.605620    2546 scope.go:117] "RemoveContainer" containerID="39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0"
Jul 17 22:03:12.606049 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:12.605797    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl_odh-ai-gateway-infra(1bbc2a2e-9c62-4f82-9dae-5e212630af1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c"
Jul 17 22:03:20.786938 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:20.786905    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:20.789927 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:20.789909    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:20.800171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:20.800143    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:20.909978 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:20.909943    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7\" (UID: \"aa5a5078-d13f-4445-abef-f862b6ab9ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:21.010592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.010561    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7\" (UID: \"aa5a5078-d13f-4445-abef-f862b6ab9ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:21.011363 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.011334    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7\" (UID: \"aa5a5078-d13f-4445-abef-f862b6ab9ee6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:21.090079 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.090004    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:03:21.093733 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.093703    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.096299 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.096277    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-592f58-openshift-default-dockercfg-jkpq8\""
Jul 17 22:03:21.100921 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.100906    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:21.105295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.105269    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:03:21.212346 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212309    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212504 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212430    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212504 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212497    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212617 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212559    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212617 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212605    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212633    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212656    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dhhh\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212693    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.212917 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.212800    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.247612 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.247578    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:21.249470 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:21.249435    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa5a5078_d13f_4445_abef_f862b6ab9ee6.slice/crio-ec706d6cc4dc3b8af8eb54c734f326f448c0472c90d3b654ee6b90699496ad49 WatchSource:0}: Error finding container ec706d6cc4dc3b8af8eb54c734f326f448c0472c90d3b654ee6b90699496ad49: Status 404 returned error can't find the container with id ec706d6cc4dc3b8af8eb54c734f326f448c0472c90d3b654ee6b90699496ad49
Jul 17 22:03:21.313835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.313812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.313847    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.313868    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.313991    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314028    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314054    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314077    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dhhh\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314116    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314493 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314232    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314571    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314779 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314757    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.314873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.314857    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.315315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.315294    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.316287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.316268    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.316435 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.316419    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.322367 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.322348    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.322742 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.322705    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dhhh\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh\") pod \"e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") " pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.406560 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.406481    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:21.552280 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.552251    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:03:21.553505 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:21.553476    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c9c912a_5de1_4201_9eba_57cd5a10ca81.slice/crio-fa104fec618cd5a1d90f49ccfcf1e3fd576e0fc76c6de51165e93e46f8bd02a0 WatchSource:0}: Error finding container fa104fec618cd5a1d90f49ccfcf1e3fd576e0fc76c6de51165e93e46f8bd02a0: Status 404 returned error can't find the container with id fa104fec618cd5a1d90f49ccfcf1e3fd576e0fc76c6de51165e93e46f8bd02a0
Jul 17 22:03:21.555604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.555572    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:21.555675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.555637    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:21.555675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.555664    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:21.661294 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.661192    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerStarted","Data":"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"}
Jul 17 22:03:21.661532 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.661511    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerStarted","Data":"fa104fec618cd5a1d90f49ccfcf1e3fd576e0fc76c6de51165e93e46f8bd02a0"}
Jul 17 22:03:21.662140 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.662113    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerStarted","Data":"e8d2052e5c626d3dd84e6a2c0c27836c9e7e9f79e3592d00a92c66cc3025e465"}
Jul 17 22:03:21.662271 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.662166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerStarted","Data":"ec706d6cc4dc3b8af8eb54c734f326f448c0472c90d3b654ee6b90699496ad49"}
Jul 17 22:03:21.683217 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.683176    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podStartSLOduration=0.683158161 podStartE2EDuration="683.158161ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:21.680412994 +0000 UTC m=+2777.660783475" watchObservedRunningTime="2026-07-17 22:03:21.683158161 +0000 UTC m=+2777.663528638"
Jul 17 22:03:21.695809 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:21.695765    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" podStartSLOduration=1.695751859 podStartE2EDuration="1.695751859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:21.694288611 +0000 UTC m=+2777.674659089" watchObservedRunningTime="2026-07-17 22:03:21.695751859 +0000 UTC m=+2777.676122336"
Jul 17 22:03:22.406996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.406919    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:22.408331 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.408306    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Jul 17 22:03:22.408441 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.408356    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Jul 17 22:03:22.668146 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.668074    2546 generic.go:356] "Generic (PLEG): container finished" podID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerID="e8d2052e5c626d3dd84e6a2c0c27836c9e7e9f79e3592d00a92c66cc3025e465" exitCode=6
Jul 17 22:03:22.668146 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.668142    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerDied","Data":"e8d2052e5c626d3dd84e6a2c0c27836c9e7e9f79e3592d00a92c66cc3025e465"}
Jul 17 22:03:22.668315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:22.668306    2546 scope.go:117] "RemoveContainer" containerID="e8d2052e5c626d3dd84e6a2c0c27836c9e7e9f79e3592d00a92c66cc3025e465"
Jul 17 22:03:23.410961 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.410936    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:23.530750 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.530680    2546 scope.go:117] "RemoveContainer" containerID="b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb"
Jul 17 22:03:23.675370 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.675346    2546 generic.go:356] "Generic (PLEG): container finished" podID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerID="4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d" exitCode=6
Jul 17 22:03:23.675492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.675423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerDied","Data":"4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d"}
Jul 17 22:03:23.675492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.675480    2546 scope.go:117] "RemoveContainer" containerID="e8d2052e5c626d3dd84e6a2c0c27836c9e7e9f79e3592d00a92c66cc3025e465"
Jul 17 22:03:23.675669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.675647    2546 scope.go:117] "RemoveContainer" containerID="4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d"
Jul 17 22:03:23.675880 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:23.675857    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7_odh-ai-gateway-infra(aa5a5078-d13f-4445-abef-f862b6ab9ee6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6"
Jul 17 22:03:23.676003 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.675959    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:23.676932 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:23.676915    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:24.682466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:24.682390    2546 generic.go:356] "Generic (PLEG): container finished" podID="c3bbf334-b2b3-432d-8792-4d052991c991" containerID="88611ee45720ced7fd43777296015d109cd7caeb131a5d99e24dc60c5d919cf6" exitCode=6
Jul 17 22:03:24.682956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:24.682465    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" event={"ID":"c3bbf334-b2b3-432d-8792-4d052991c991","Type":"ContainerDied","Data":"88611ee45720ced7fd43777296015d109cd7caeb131a5d99e24dc60c5d919cf6"}
Jul 17 22:03:24.682956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:24.682531    2546 scope.go:117] "RemoveContainer" containerID="b2f7ca12da0e84a8aaa00a2a08dbb1645505a58f769e4e56180e294cb661c6cb"
Jul 17 22:03:24.682956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:24.682807    2546 scope.go:117] "RemoveContainer" containerID="88611ee45720ced7fd43777296015d109cd7caeb131a5d99e24dc60c5d919cf6"
Jul 17 22:03:24.683123 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:24.682992    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6_odh-ai-gateway-infra(c3bbf334-b2b3-432d-8792-4d052991c991)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" podUID="c3bbf334-b2b3-432d-8792-4d052991c991"
Jul 17 22:03:24.684341 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:24.684324    2546 scope.go:117] "RemoveContainer" containerID="4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d"
Jul 17 22:03:24.684527 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:24.684506    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7_odh-ai-gateway-infra(aa5a5078-d13f-4445-abef-f862b6ab9ee6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6"
Jul 17 22:03:25.540418 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.540383    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:25.544773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.544757    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:25.555077 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.555047    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:25.652828 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.652796    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr\" (UID: \"7e9673f3-6e18-4e52-95c8-38f1da645475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:25.711576 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.711549    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:25.754248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.754208    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr\" (UID: \"7e9673f3-6e18-4e52-95c8-38f1da645475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:25.754853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.754831    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr\" (UID: \"7e9673f3-6e18-4e52-95c8-38f1da645475\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:25.845068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.845045    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:25.863586 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.863552    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:25.955818 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.955786    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca\") pod \"c3bbf334-b2b3-432d-8792-4d052991c991\" (UID: \"c3bbf334-b2b3-432d-8792-4d052991c991\") "
Jul 17 22:03:25.956227 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.956201    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3bbf334-b2b3-432d-8792-4d052991c991" (UID: "c3bbf334-b2b3-432d-8792-4d052991c991"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:25.998356 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:25.998325    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:25.999963 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:25.999938    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e9673f3_6e18_4e52_95c8_38f1da645475.slice/crio-5fd5714b903717d61c129838d4acb43f4081a423e2f67bc4f086c8725c292097 WatchSource:0}: Error finding container 5fd5714b903717d61c129838d4acb43f4081a423e2f67bc4f086c8725c292097: Status 404 returned error can't find the container with id 5fd5714b903717d61c129838d4acb43f4081a423e2f67bc4f086c8725c292097
Jul 17 22:03:26.056793 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.056773    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3bbf334-b2b3-432d-8792-4d052991c991-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:03:26.531419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.531394    2546 scope.go:117] "RemoveContainer" containerID="39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0"
Jul 17 22:03:26.698007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.697977    2546 generic.go:356] "Generic (PLEG): container finished" podID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerID="4644f35594d3e2a239a2aef1141598c87fe174645b08a67717a5a448109ddd2a" exitCode=6
Jul 17 22:03:26.698124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.698058    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" event={"ID":"7e9673f3-6e18-4e52-95c8-38f1da645475","Type":"ContainerDied","Data":"4644f35594d3e2a239a2aef1141598c87fe174645b08a67717a5a448109ddd2a"}
Jul 17 22:03:26.698124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.698106    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" event={"ID":"7e9673f3-6e18-4e52-95c8-38f1da645475","Type":"ContainerStarted","Data":"5fd5714b903717d61c129838d4acb43f4081a423e2f67bc4f086c8725c292097"}
Jul 17 22:03:26.698235 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.698208    2546 scope.go:117] "RemoveContainer" containerID="4644f35594d3e2a239a2aef1141598c87fe174645b08a67717a5a448109ddd2a"
Jul 17 22:03:26.699340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.699328    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"
Jul 17 22:03:26.699426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.699344    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6" event={"ID":"c3bbf334-b2b3-432d-8792-4d052991c991","Type":"ContainerDied","Data":"c6666f043c8551c1cd2dd6e94ca33e2d502063c4eddf494f9720180ca99d33f3"}
Jul 17 22:03:26.699426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.699387    2546 scope.go:117] "RemoveContainer" containerID="88611ee45720ced7fd43777296015d109cd7caeb131a5d99e24dc60c5d919cf6"
Jul 17 22:03:26.728058 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.728040    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:26.733755 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:26.733735    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-zzzf6"]
Jul 17 22:03:27.707582 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.707547    2546 generic.go:356] "Generic (PLEG): container finished" podID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerID="b78858a07e39f64e4982d0967b40d90c4baf6dddd8f06d4bfd7833719e9fb5e9" exitCode=6
Jul 17 22:03:27.707802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.707628    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" event={"ID":"1bbc2a2e-9c62-4f82-9dae-5e212630af1c","Type":"ContainerDied","Data":"b78858a07e39f64e4982d0967b40d90c4baf6dddd8f06d4bfd7833719e9fb5e9"}
Jul 17 22:03:27.707802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.707691    2546 scope.go:117] "RemoveContainer" containerID="39d63d76c155c2e518c5f4ef137aec9630d27f51e89bc87c744598a1473e21c0"
Jul 17 22:03:27.708034 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.708016    2546 scope.go:117] "RemoveContainer" containerID="b78858a07e39f64e4982d0967b40d90c4baf6dddd8f06d4bfd7833719e9fb5e9"
Jul 17 22:03:27.708269 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:27.708240    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl_odh-ai-gateway-infra(1bbc2a2e-9c62-4f82-9dae-5e212630af1c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c"
Jul 17 22:03:27.709558 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.709537    2546 generic.go:356] "Generic (PLEG): container finished" podID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerID="91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8" exitCode=6
Jul 17 22:03:27.709644 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.709568    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" event={"ID":"7e9673f3-6e18-4e52-95c8-38f1da645475","Type":"ContainerDied","Data":"91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8"}
Jul 17 22:03:27.709774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.709760    2546 scope.go:117] "RemoveContainer" containerID="91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8"
Jul 17 22:03:27.709939 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:27.709921    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr_odh-ai-gateway-infra(7e9673f3-6e18-4e52-95c8-38f1da645475)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475"
Jul 17 22:03:27.719431 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:27.719254    2546 scope.go:117] "RemoveContainer" containerID="4644f35594d3e2a239a2aef1141598c87fe174645b08a67717a5a448109ddd2a"
Jul 17 22:03:28.536476 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.536443    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" path="/var/lib/kubelet/pods/c3bbf334-b2b3-432d-8792-4d052991c991/volumes"
Jul 17 22:03:28.649999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.649970    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:03:28.650309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650296    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650311    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650323    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650329    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650338    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650346    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650409    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.650521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.650421    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:28.654579 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.654559    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.657926 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.657903    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-6be22b-openshift-default-dockercfg-tvtrj\""
Jul 17 22:03:28.666851 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.666833    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:03:28.717395 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.717374    2546 scope.go:117] "RemoveContainer" containerID="91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8"
Jul 17 22:03:28.717564 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:28.717543    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr_odh-ai-gateway-infra(7e9673f3-6e18-4e52-95c8-38f1da645475)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475"
Jul 17 22:03:28.739916 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.739889    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:28.779765 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.779733    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.779949 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.779806    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.779949 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.779871    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780060 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.779967    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvsvp\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780060 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.780008    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780060 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.780026    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.780111    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.780135    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.780213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.780152    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.875827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.875804    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:28.880996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.880972    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881112 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881016    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881112 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881046    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881125    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvsvp\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881174    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881206    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881382 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881259    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881382 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881285    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881382 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881315    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881533 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881386    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881644 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881618    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881784 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881748    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881784 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881761    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.881873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.881823    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.883537 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.883517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.883655 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.883637    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.891483 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.891435    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvsvp\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.891483 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.891484    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token\") pod \"e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") " pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.964873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.964840    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:28.982067 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.982030    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca\") pod \"1bbc2a2e-9c62-4f82-9dae-5e212630af1c\" (UID: \"1bbc2a2e-9c62-4f82-9dae-5e212630af1c\") "
Jul 17 22:03:28.982345 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:28.982323    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bbc2a2e-9c62-4f82-9dae-5e212630af1c" (UID: "1bbc2a2e-9c62-4f82-9dae-5e212630af1c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:29.083136 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.083108    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bbc2a2e-9c62-4f82-9dae-5e212630af1c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:03:29.109559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.109530    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:03:29.111024 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:29.111001    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55fc2ea_66ba_429d_8bd3_546c05fd5f67.slice/crio-f8fc02f5ef4d469597307561d73191b368e9dd960e0e20eb5bc64f0d2a06f1d4 WatchSource:0}: Error finding container f8fc02f5ef4d469597307561d73191b368e9dd960e0e20eb5bc64f0d2a06f1d4: Status 404 returned error can't find the container with id f8fc02f5ef4d469597307561d73191b368e9dd960e0e20eb5bc64f0d2a06f1d4
Jul 17 22:03:29.113260 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.113233    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:29.113318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.113296    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:29.113369 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.113332    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:29.723789 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.723751    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerStarted","Data":"0c67bcc98cd781d2089ba655ab8ab8597873c0b9e6c7c8abd75d8fa779cb48d3"}
Jul 17 22:03:29.724273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.723815    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerStarted","Data":"f8fc02f5ef4d469597307561d73191b368e9dd960e0e20eb5bc64f0d2a06f1d4"}
Jul 17 22:03:29.725159 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.725136    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"
Jul 17 22:03:29.725159 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.725145    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl" event={"ID":"1bbc2a2e-9c62-4f82-9dae-5e212630af1c","Type":"ContainerDied","Data":"841a535228f7e0fdc645c0a3b2823d90d395b30c99a672255dd0343de2a94143"}
Jul 17 22:03:29.725355 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.725201    2546 scope.go:117] "RemoveContainer" containerID="b78858a07e39f64e4982d0967b40d90c4baf6dddd8f06d4bfd7833719e9fb5e9"
Jul 17 22:03:29.742952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.742816    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podStartSLOduration=1.7427987200000001 podStartE2EDuration="1.74279872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:29.742373154 +0000 UTC m=+2785.722743641" watchObservedRunningTime="2026-07-17 22:03:29.74279872 +0000 UTC m=+2785.723169210"
Jul 17 22:03:29.758645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.758615    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:29.761909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.761886    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cwrzl"]
Jul 17 22:03:29.965446 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:29.965410    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:30.536957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:30.536923    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" path="/var/lib/kubelet/pods/1bbc2a2e-9c62-4f82-9dae-5e212630af1c/volumes"
Jul 17 22:03:30.970176 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:30.970102    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:31.735523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:31.735483    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:31.736367 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:31.736350    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:38.531491 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:38.531455    2546 scope.go:117] "RemoveContainer" containerID="4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d"
Jul 17 22:03:39.000830 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.000797    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": read tcp 10.133.0.2:55286->10.133.0.117:15021: read: connection reset by peer" start-of-body=
Jul 17 22:03:39.000991 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.000859    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": read tcp 10.133.0.2:55286->10.133.0.117:15021: read: connection reset by peer"
Jul 17 22:03:39.780673 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.780626    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerID="d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54" exitCode=0
Jul 17 22:03:39.781194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.780742    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerDied","Data":"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"}
Jul 17 22:03:39.781271 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.781212    2546 scope.go:117] "RemoveContainer" containerID="d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"
Jul 17 22:03:39.782836 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.782793    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:39.782978 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.782882    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:39.782978 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.782923    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:39.785867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.785702    2546 generic.go:356] "Generic (PLEG): container finished" podID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerID="6db61113f0ca2d1ee97f108534e545f9d9a1e4fd9ffffba96c126568f3cdfe19" exitCode=6
Jul 17 22:03:39.785867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.785755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerDied","Data":"6db61113f0ca2d1ee97f108534e545f9d9a1e4fd9ffffba96c126568f3cdfe19"}
Jul 17 22:03:39.785867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.785816    2546 scope.go:117] "RemoveContainer" containerID="4287dd402f5ce218c657d6df3e242c6a12cce767e76d14fed36c1c60fe6e6b1d"
Jul 17 22:03:39.786304 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:39.786111    2546 scope.go:117] "RemoveContainer" containerID="6db61113f0ca2d1ee97f108534e545f9d9a1e4fd9ffffba96c126568f3cdfe19"
Jul 17 22:03:39.786368 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:39.786297    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7_odh-ai-gateway-infra(aa5a5078-d13f-4445-abef-f862b6ab9ee6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6"
Jul 17 22:03:40.816305 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:40.816268    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerStarted","Data":"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"}
Jul 17 22:03:40.840995 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:40.840946    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:41.069592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.069266    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:41.204063 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.203999    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca\") pod \"aa5a5078-d13f-4445-abef-f862b6ab9ee6\" (UID: \"aa5a5078-d13f-4445-abef-f862b6ab9ee6\") "
Jul 17 22:03:41.204745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.204690    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa5a5078-d13f-4445-abef-f862b6ab9ee6" (UID: "aa5a5078-d13f-4445-abef-f862b6ab9ee6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:41.305744 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.305679    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa5a5078-d13f-4445-abef-f862b6ab9ee6-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:03:41.407048 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.406933    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:41.407465 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.407437    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Jul 17 22:03:41.407611 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.407497    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Jul 17 22:03:41.825110 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.825078    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"
Jul 17 22:03:41.825616 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.825121    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7" event={"ID":"aa5a5078-d13f-4445-abef-f862b6ab9ee6","Type":"ContainerDied","Data":"ec706d6cc4dc3b8af8eb54c734f326f448c0472c90d3b654ee6b90699496ad49"}
Jul 17 22:03:41.825748 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.825636    2546 scope.go:117] "RemoveContainer" containerID="6db61113f0ca2d1ee97f108534e545f9d9a1e4fd9ffffba96c126568f3cdfe19"
Jul 17 22:03:41.850865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.850827    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:41.857795 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:41.857714    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-8lqm7"]
Jul 17 22:03:42.407125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:42.407089    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Jul 17 22:03:42.407467 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:42.407417    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Jul 17 22:03:42.536798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:42.536771    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" path="/var/lib/kubelet/pods/aa5a5078-d13f-4445-abef-f862b6ab9ee6/volumes"
Jul 17 22:03:43.406956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:43.406878    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Jul 17 22:03:43.406956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:43.406929    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Jul 17 22:03:43.531736 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:43.531690    2546 scope.go:117] "RemoveContainer" containerID="91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8"
Jul 17 22:03:44.406921 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.406886    2546 patch_prober.go:28] interesting pod/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused" start-of-body=
Jul 17 22:03:44.407106 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.406952    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.117:15021/healthz/ready\": dial tcp 10.133.0.117:15021: connect: connection refused"
Jul 17 22:03:44.843488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.843457    2546 generic.go:356] "Generic (PLEG): container finished" podID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerID="a4f57a36a2d0fd7994e17a3d8de820f9291a44706b388815053196b280b4b9e6" exitCode=6
Jul 17 22:03:44.843651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.843531    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" event={"ID":"7e9673f3-6e18-4e52-95c8-38f1da645475","Type":"ContainerDied","Data":"a4f57a36a2d0fd7994e17a3d8de820f9291a44706b388815053196b280b4b9e6"}
Jul 17 22:03:44.843651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.843585    2546 scope.go:117] "RemoveContainer" containerID="91e18eda10fb03de6a2f356fbb75d262caa724e4dc5a0b63a662325487e149d8"
Jul 17 22:03:44.843968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:44.843948    2546 scope.go:117] "RemoveContainer" containerID="a4f57a36a2d0fd7994e17a3d8de820f9291a44706b388815053196b280b4b9e6"
Jul 17 22:03:44.844190 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:03:44.844166    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr_odh-ai-gateway-infra(7e9673f3-6e18-4e52-95c8-38f1da645475)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475"
Jul 17 22:03:45.410986 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:45.410954    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:45.411387 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:45.411331    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:45.412006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:45.411987    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:03:45.871258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:45.871221    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:46.004802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.004780    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:46.152949 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.152921    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca\") pod \"7e9673f3-6e18-4e52-95c8-38f1da645475\" (UID: \"7e9673f3-6e18-4e52-95c8-38f1da645475\") "
Jul 17 22:03:46.154608 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.153236    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e9673f3-6e18-4e52-95c8-38f1da645475" (UID: "7e9673f3-6e18-4e52-95c8-38f1da645475"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:03:46.254428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.254401    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e9673f3-6e18-4e52-95c8-38f1da645475-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:03:46.737477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.737448    2546 patch_prober.go:28] interesting pod/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:03:46.737870 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.737508    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:03:46.855698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.855641    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"
Jul 17 22:03:46.855698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.855675    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr" event={"ID":"7e9673f3-6e18-4e52-95c8-38f1da645475","Type":"ContainerDied","Data":"5fd5714b903717d61c129838d4acb43f4081a423e2f67bc4f086c8725c292097"}
Jul 17 22:03:46.855927 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.855763    2546 scope.go:117] "RemoveContainer" containerID="a4f57a36a2d0fd7994e17a3d8de820f9291a44706b388815053196b280b4b9e6"
Jul 17 22:03:46.857797 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.857771    2546 generic.go:356] "Generic (PLEG): container finished" podID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerID="0c67bcc98cd781d2089ba655ab8ab8597873c0b9e6c7c8abd75d8fa779cb48d3" exitCode=0
Jul 17 22:03:46.857912 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.857875    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerDied","Data":"0c67bcc98cd781d2089ba655ab8ab8597873c0b9e6c7c8abd75d8fa779cb48d3"}
Jul 17 22:03:46.858258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.858237    2546 scope.go:117] "RemoveContainer" containerID="0c67bcc98cd781d2089ba655ab8ab8597873c0b9e6c7c8abd75d8fa779cb48d3"
Jul 17 22:03:46.859315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.859288    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:46.859422 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.859369    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:46.859488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.859422    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:03:46.895009 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.894981    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:46.897971 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:46.897947    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-kfzxr"]
Jul 17 22:03:47.873663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:47.873618    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerStarted","Data":"26f379f4b9fc7c59c79463a241ca98dfdfc2249cb261bad20b441189e666c354"}
Jul 17 22:03:47.967121 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:47.967073    2546 patch_prober.go:28] interesting pod/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:03:47.967326 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:47.967155    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:03:47.967502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:47.966646    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:48.536247 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:48.536211    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" path="/var/lib/kubelet/pods/7e9673f3-6e18-4e52-95c8-38f1da645475/volumes"
Jul 17 22:03:48.965586 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:48.965498    2546 patch_prober.go:28] interesting pod/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:03:48.966165 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:48.966134    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:03:49.966071 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:49.966035    2546 patch_prober.go:28] interesting pod/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:03:49.966668 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:49.966634    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:03:50.965951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:50.965514    2546 patch_prober.go:28] interesting pod/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 17 22:03:50.965951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:50.965573    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 17 22:03:51.968770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:51.968742    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:51.969177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:51.969134    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:51.969937 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:51.969920    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:03:58.076014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.075985    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076269    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076279    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076288    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076294    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076308    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076313    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076321    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076326    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076333    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076338    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076349    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076353    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076399    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076406    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076411    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076416    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076421    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076426    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076432    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076437    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:03:58.076820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.076442    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3bbf334-b2b3-432d-8792-4d052991c991" containerName="revoke-keys"
Jul 17 22:03:58.078999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.078982    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:03:58.081227 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.081207    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:03:58.086712 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.086530    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:03:58.160736 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.160689    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd\" (UID: \"b4565678-0a97-4515-9006-13e3ba619d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:03:58.261429 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.261403    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd\" (UID: \"b4565678-0a97-4515-9006-13e3ba619d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:03:58.262014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.261997    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd\" (UID: \"b4565678-0a97-4515-9006-13e3ba619d37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:03:58.390320 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.390252    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:03:58.517164 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.517142    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:03:58.519208 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:03:58.519170    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4565678_0a97_4515_9006_13e3ba619d37.slice/crio-7b1d7fed3519eb6d3b44e5cc256e831af95472da88e27c3ae3fa3cccdc385ece WatchSource:0}: Error finding container 7b1d7fed3519eb6d3b44e5cc256e831af95472da88e27c3ae3fa3cccdc385ece: Status 404 returned error can't find the container with id 7b1d7fed3519eb6d3b44e5cc256e831af95472da88e27c3ae3fa3cccdc385ece
Jul 17 22:03:58.938297 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.938214    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerStarted","Data":"ed0f08fb3c610659868ce756a58fbf133bcf609465b165b2313c6c7591a4ad9b"}
Jul 17 22:03:58.938297 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.938264    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerStarted","Data":"7b1d7fed3519eb6d3b44e5cc256e831af95472da88e27c3ae3fa3cccdc385ece"}
Jul 17 22:03:58.955201 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:58.955150    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" podStartSLOduration=0.955135965 podStartE2EDuration="955.135965ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:03:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:03:58.953102656 +0000 UTC m=+2814.933473135" watchObservedRunningTime="2026-07-17 22:03:58.955135965 +0000 UTC m=+2814.935506447"
Jul 17 22:03:59.944420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:59.944385    2546 generic.go:356] "Generic (PLEG): container finished" podID="b4565678-0a97-4515-9006-13e3ba619d37" containerID="ed0f08fb3c610659868ce756a58fbf133bcf609465b165b2313c6c7591a4ad9b" exitCode=6
Jul 17 22:03:59.944825 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:59.944438    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerDied","Data":"ed0f08fb3c610659868ce756a58fbf133bcf609465b165b2313c6c7591a4ad9b"}
Jul 17 22:03:59.944825 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:03:59.944671    2546 scope.go:117] "RemoveContainer" containerID="ed0f08fb3c610659868ce756a58fbf133bcf609465b165b2313c6c7591a4ad9b"
Jul 17 22:04:00.951528 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:00.951448    2546 generic.go:356] "Generic (PLEG): container finished" podID="b4565678-0a97-4515-9006-13e3ba619d37" containerID="8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb" exitCode=6
Jul 17 22:04:00.951528 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:00.951512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerDied","Data":"8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb"}
Jul 17 22:04:00.952002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:00.951555    2546 scope.go:117] "RemoveContainer" containerID="ed0f08fb3c610659868ce756a58fbf133bcf609465b165b2313c6c7591a4ad9b"
Jul 17 22:04:00.952002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:00.951734    2546 scope.go:117] "RemoveContainer" containerID="8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb"
Jul 17 22:04:00.952090 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:00.952008    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd_odh-ai-gateway-infra(b4565678-0a97-4515-9006-13e3ba619d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" podUID="b4565678-0a97-4515-9006-13e3ba619d37"
Jul 17 22:04:01.958272 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:01.958243    2546 scope.go:117] "RemoveContainer" containerID="8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb"
Jul 17 22:04:01.958658 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:01.958475    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd_odh-ai-gateway-infra(b4565678-0a97-4515-9006-13e3ba619d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" podUID="b4565678-0a97-4515-9006-13e3ba619d37"
Jul 17 22:04:08.998557 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.998514    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.998975    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.998993    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bbc2a2e-9c62-4f82-9dae-5e212630af1c" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.999015    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.999023    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e9673f3-6e18-4e52-95c8-38f1da645475" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.999041    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.999050    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:04:08.999184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:08.999127    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa5a5078-d13f-4445-abef-f862b6ab9ee6" containerName="revoke-keys"
Jul 17 22:04:09.002234 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.002213    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:09.009015 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.008992    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:09.044743 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.044701    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96\" (UID: \"dcde46a8-ee85-45b0-a9d0-3a1a310d1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:09.145137 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.145109    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96\" (UID: \"dcde46a8-ee85-45b0-a9d0-3a1a310d1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:09.145688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.145669    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96\" (UID: \"dcde46a8-ee85-45b0-a9d0-3a1a310d1689\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:09.313464 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.313410    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:09.453293 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:09.453261    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:09.454852 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:04:09.454827    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcde46a8_ee85_45b0_a9d0_3a1a310d1689.slice/crio-f80bcd6f941530a4f08783cee4133e0dee11ef97e7c645c35b8e5900d6f7a799 WatchSource:0}: Error finding container f80bcd6f941530a4f08783cee4133e0dee11ef97e7c645c35b8e5900d6f7a799: Status 404 returned error can't find the container with id f80bcd6f941530a4f08783cee4133e0dee11ef97e7c645c35b8e5900d6f7a799
Jul 17 22:04:10.000123 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:10.000089    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerStarted","Data":"fdaefc0f1358f5d67db79ec00b63ea00874752d5cf33fb1e921523495bd3e985"}
Jul 17 22:04:10.000485 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:10.000140    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerStarted","Data":"f80bcd6f941530a4f08783cee4133e0dee11ef97e7c645c35b8e5900d6f7a799"}
Jul 17 22:04:10.017074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:10.017030    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" podStartSLOduration=2.017017123 podStartE2EDuration="2.017017123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:04:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:04:10.015376485 +0000 UTC m=+2825.995746973" watchObservedRunningTime="2026-07-17 22:04:10.017017123 +0000 UTC m=+2825.997387626"
Jul 17 22:04:11.007713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:11.007674    2546 generic.go:356] "Generic (PLEG): container finished" podID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerID="fdaefc0f1358f5d67db79ec00b63ea00874752d5cf33fb1e921523495bd3e985" exitCode=6
Jul 17 22:04:11.007713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:11.007709    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerDied","Data":"fdaefc0f1358f5d67db79ec00b63ea00874752d5cf33fb1e921523495bd3e985"}
Jul 17 22:04:11.008186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:11.008003    2546 scope.go:117] "RemoveContainer" containerID="fdaefc0f1358f5d67db79ec00b63ea00874752d5cf33fb1e921523495bd3e985"
Jul 17 22:04:12.015038 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:12.015003    2546 generic.go:356] "Generic (PLEG): container finished" podID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerID="d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44" exitCode=6
Jul 17 22:04:12.015461 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:12.015087    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerDied","Data":"d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44"}
Jul 17 22:04:12.015461 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:12.015146    2546 scope.go:117] "RemoveContainer" containerID="fdaefc0f1358f5d67db79ec00b63ea00874752d5cf33fb1e921523495bd3e985"
Jul 17 22:04:12.015461 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:12.015369    2546 scope.go:117] "RemoveContainer" containerID="d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44"
Jul 17 22:04:12.015589 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:12.015568    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96_odh-ai-gateway-infra(dcde46a8-ee85-45b0-a9d0-3a1a310d1689)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689"
Jul 17 22:04:13.021865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:13.021835    2546 scope.go:117] "RemoveContainer" containerID="d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44"
Jul 17 22:04:13.022259 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:13.022008    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96_odh-ai-gateway-infra(dcde46a8-ee85-45b0-a9d0-3a1a310d1689)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689"
Jul 17 22:04:13.530968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:13.530942    2546 scope.go:117] "RemoveContainer" containerID="8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb"
Jul 17 22:04:14.028221 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:14.028184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerStarted","Data":"f9c1b4e39ca2f9e0c1a11899dd6fb1970cb705b4442799af089dd969799e38c4"}
Jul 17 22:04:15.034847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:15.034810    2546 generic.go:356] "Generic (PLEG): container finished" podID="b4565678-0a97-4515-9006-13e3ba619d37" containerID="f9c1b4e39ca2f9e0c1a11899dd6fb1970cb705b4442799af089dd969799e38c4" exitCode=6
Jul 17 22:04:15.034847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:15.034846    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerDied","Data":"f9c1b4e39ca2f9e0c1a11899dd6fb1970cb705b4442799af089dd969799e38c4"}
Jul 17 22:04:15.035361 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:15.034895    2546 scope.go:117] "RemoveContainer" containerID="8ea756d3284f5b2cf17e7c3d7bf9740a97a9f3013b273d98d93efbd2508c8acb"
Jul 17 22:04:15.035361 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:15.035182    2546 scope.go:117] "RemoveContainer" containerID="f9c1b4e39ca2f9e0c1a11899dd6fb1970cb705b4442799af089dd969799e38c4"
Jul 17 22:04:15.035361 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:15.035354    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd_odh-ai-gateway-infra(b4565678-0a97-4515-9006-13e3ba619d37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" podUID="b4565678-0a97-4515-9006-13e3ba619d37"
Jul 17 22:04:15.053993 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:15.053969    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:04:16.168532 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:16.168511    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:04:16.304804 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:16.304740    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca\") pod \"b4565678-0a97-4515-9006-13e3ba619d37\" (UID: \"b4565678-0a97-4515-9006-13e3ba619d37\") "
Jul 17 22:04:16.304942 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:16.304923    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4565678-0a97-4515-9006-13e3ba619d37" (UID: "b4565678-0a97-4515-9006-13e3ba619d37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:16.405815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:16.405791    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4565678-0a97-4515-9006-13e3ba619d37-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:04:17.047313 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:17.047282    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"
Jul 17 22:04:17.047518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:17.047314    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd" event={"ID":"b4565678-0a97-4515-9006-13e3ba619d37","Type":"ContainerDied","Data":"7b1d7fed3519eb6d3b44e5cc256e831af95472da88e27c3ae3fa3cccdc385ece"}
Jul 17 22:04:17.047518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:17.047369    2546 scope.go:117] "RemoveContainer" containerID="f9c1b4e39ca2f9e0c1a11899dd6fb1970cb705b4442799af089dd969799e38c4"
Jul 17 22:04:17.066187 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:17.066162    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:04:17.068741 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:17.068708    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-gmmrd"]
Jul 17 22:04:18.536392 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:18.536353    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4565678-0a97-4515-9006-13e3ba619d37" path="/var/lib/kubelet/pods/b4565678-0a97-4515-9006-13e3ba619d37/volumes"
Jul 17 22:04:20.836108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836074    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836377    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836387    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836401    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836407    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836415    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836420    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836487    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.836509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.836494    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:20.841019 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.840993    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:20.847660 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.847637    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:20.939037 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:20.938999    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr\" (UID: \"ed3f268b-f579-43a3-8c24-38a362b229fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:21.039593 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:21.039565    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr\" (UID: \"ed3f268b-f579-43a3-8c24-38a362b229fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:21.040197 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:21.040174    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr\" (UID: \"ed3f268b-f579-43a3-8c24-38a362b229fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:21.153444 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:21.153385    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:21.301976 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:21.301955    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:21.303890 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:04:21.303866    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded3f268b_f579_43a3_8c24_38a362b229fd.slice/crio-131fb824758db81affad45120291048e8679a9dd19bb4f84e9be52c28d8b67a0 WatchSource:0}: Error finding container 131fb824758db81affad45120291048e8679a9dd19bb4f84e9be52c28d8b67a0: Status 404 returned error can't find the container with id 131fb824758db81affad45120291048e8679a9dd19bb4f84e9be52c28d8b67a0
Jul 17 22:04:22.075640 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:22.075608    2546 generic.go:356] "Generic (PLEG): container finished" podID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerID="7332fb898b59b8c28201c692e3def843019a5d2e0a38a67bf9573cdc51ac951f" exitCode=6
Jul 17 22:04:22.076040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:22.075694    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" event={"ID":"ed3f268b-f579-43a3-8c24-38a362b229fd","Type":"ContainerDied","Data":"7332fb898b59b8c28201c692e3def843019a5d2e0a38a67bf9573cdc51ac951f"}
Jul 17 22:04:22.076040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:22.075761    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" event={"ID":"ed3f268b-f579-43a3-8c24-38a362b229fd","Type":"ContainerStarted","Data":"131fb824758db81affad45120291048e8679a9dd19bb4f84e9be52c28d8b67a0"}
Jul 17 22:04:22.076040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:22.075897    2546 scope.go:117] "RemoveContainer" containerID="7332fb898b59b8c28201c692e3def843019a5d2e0a38a67bf9573cdc51ac951f"
Jul 17 22:04:23.083768 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:23.083695    2546 generic.go:356] "Generic (PLEG): container finished" podID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerID="cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c" exitCode=6
Jul 17 22:04:23.084161 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:23.083795    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" event={"ID":"ed3f268b-f579-43a3-8c24-38a362b229fd","Type":"ContainerDied","Data":"cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c"}
Jul 17 22:04:23.084161 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:23.083869    2546 scope.go:117] "RemoveContainer" containerID="7332fb898b59b8c28201c692e3def843019a5d2e0a38a67bf9573cdc51ac951f"
Jul 17 22:04:23.084161 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:23.084092    2546 scope.go:117] "RemoveContainer" containerID="cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c"
Jul 17 22:04:23.084335 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:23.084313    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr_odh-ai-gateway-infra(ed3f268b-f579-43a3-8c24-38a362b229fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd"
Jul 17 22:04:24.090873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:24.090847    2546 scope.go:117] "RemoveContainer" containerID="cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c"
Jul 17 22:04:24.091247 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:24.091017    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr_odh-ai-gateway-infra(ed3f268b-f579-43a3-8c24-38a362b229fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd"
Jul 17 22:04:25.600361 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.600328    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:25.600820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.600801    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4565678-0a97-4515-9006-13e3ba619d37" containerName="revoke-keys"
Jul 17 22:04:25.605261 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.605243    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:25.611424 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.611393    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:25.676314 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.676291    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw\" (UID: \"ac5509b9-b75a-4ca6-8095-21051af90514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:25.776636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.776606    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw\" (UID: \"ac5509b9-b75a-4ca6-8095-21051af90514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:25.777234 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.777212    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw\" (UID: \"ac5509b9-b75a-4ca6-8095-21051af90514\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:25.917853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:25.917775    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:26.046978 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:26.046955    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:26.049314 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:04:26.049287    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5509b9_b75a_4ca6_8095_21051af90514.slice/crio-b2de0e5289f01ac684d2c6480e7e57bfea65d345f201004be048f64777dff2a8 WatchSource:0}: Error finding container b2de0e5289f01ac684d2c6480e7e57bfea65d345f201004be048f64777dff2a8: Status 404 returned error can't find the container with id b2de0e5289f01ac684d2c6480e7e57bfea65d345f201004be048f64777dff2a8
Jul 17 22:04:26.101713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:26.101687    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" event={"ID":"ac5509b9-b75a-4ca6-8095-21051af90514","Type":"ContainerStarted","Data":"b2de0e5289f01ac684d2c6480e7e57bfea65d345f201004be048f64777dff2a8"}
Jul 17 22:04:27.108122 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:27.108053    2546 generic.go:356] "Generic (PLEG): container finished" podID="ac5509b9-b75a-4ca6-8095-21051af90514" containerID="92050577dbacc4e86e33e963f6799ba579e597c901b274c7242f999044720d0c" exitCode=6
Jul 17 22:04:27.108122 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:27.108108    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" event={"ID":"ac5509b9-b75a-4ca6-8095-21051af90514","Type":"ContainerDied","Data":"92050577dbacc4e86e33e963f6799ba579e597c901b274c7242f999044720d0c"}
Jul 17 22:04:27.108494 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:27.108290    2546 scope.go:117] "RemoveContainer" containerID="92050577dbacc4e86e33e963f6799ba579e597c901b274c7242f999044720d0c"
Jul 17 22:04:28.114642 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:28.114610    2546 generic.go:356] "Generic (PLEG): container finished" podID="ac5509b9-b75a-4ca6-8095-21051af90514" containerID="351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214" exitCode=6
Jul 17 22:04:28.115074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:28.114702    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" event={"ID":"ac5509b9-b75a-4ca6-8095-21051af90514","Type":"ContainerDied","Data":"351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214"}
Jul 17 22:04:28.115074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:28.114768    2546 scope.go:117] "RemoveContainer" containerID="92050577dbacc4e86e33e963f6799ba579e597c901b274c7242f999044720d0c"
Jul 17 22:04:28.115074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:28.114892    2546 scope.go:117] "RemoveContainer" containerID="351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214"
Jul 17 22:04:28.115185 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:28.115092    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw_odh-ai-gateway-infra(ac5509b9-b75a-4ca6-8095-21051af90514)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" podUID="ac5509b9-b75a-4ca6-8095-21051af90514"
Jul 17 22:04:28.531838 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:28.531806    2546 scope.go:117] "RemoveContainer" containerID="d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44"
Jul 17 22:04:29.120989 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:29.120905    2546 scope.go:117] "RemoveContainer" containerID="351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214"
Jul 17 22:04:29.121402 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:29.121086    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw_odh-ai-gateway-infra(ac5509b9-b75a-4ca6-8095-21051af90514)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" podUID="ac5509b9-b75a-4ca6-8095-21051af90514"
Jul 17 22:04:29.122547 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:29.122524    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerStarted","Data":"d3eb44aa9322521213dfea503716c98e3480775c4cfa66f3f2be2bc6a479c2e8"}
Jul 17 22:04:30.129282 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:30.129250    2546 generic.go:356] "Generic (PLEG): container finished" podID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerID="d3eb44aa9322521213dfea503716c98e3480775c4cfa66f3f2be2bc6a479c2e8" exitCode=6
Jul 17 22:04:30.129750 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:30.129329    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerDied","Data":"d3eb44aa9322521213dfea503716c98e3480775c4cfa66f3f2be2bc6a479c2e8"}
Jul 17 22:04:30.129750 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:30.129398    2546 scope.go:117] "RemoveContainer" containerID="d517f3c1bceece0b23339602acbd6df649ff2af6f5b75ffc740708b4262f5f44"
Jul 17 22:04:30.129750 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:30.129697    2546 scope.go:117] "RemoveContainer" containerID="d3eb44aa9322521213dfea503716c98e3480775c4cfa66f3f2be2bc6a479c2e8"
Jul 17 22:04:30.129963 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:30.129941    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96_odh-ai-gateway-infra(dcde46a8-ee85-45b0-a9d0-3a1a310d1689)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689"
Jul 17 22:04:30.166223 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:30.166195    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:31.279558 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:31.279535    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:31.419759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:31.419665    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca\") pod \"dcde46a8-ee85-45b0-a9d0-3a1a310d1689\" (UID: \"dcde46a8-ee85-45b0-a9d0-3a1a310d1689\") "
Jul 17 22:04:31.420041 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:31.420021    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dcde46a8-ee85-45b0-a9d0-3a1a310d1689" (UID: "dcde46a8-ee85-45b0-a9d0-3a1a310d1689"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:31.520989 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:31.520964    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dcde46a8-ee85-45b0-a9d0-3a1a310d1689-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:04:32.142465 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.142431    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"
Jul 17 22:04:32.142649 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.142482    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96" event={"ID":"dcde46a8-ee85-45b0-a9d0-3a1a310d1689","Type":"ContainerDied","Data":"f80bcd6f941530a4f08783cee4133e0dee11ef97e7c645c35b8e5900d6f7a799"}
Jul 17 22:04:32.142649 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.142539    2546 scope.go:117] "RemoveContainer" containerID="d3eb44aa9322521213dfea503716c98e3480775c4cfa66f3f2be2bc6a479c2e8"
Jul 17 22:04:32.165208 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.165179    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:32.170884 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.170866    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-vsf96"]
Jul 17 22:04:32.536231 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:32.536203    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" path="/var/lib/kubelet/pods/dcde46a8-ee85-45b0-a9d0-3a1a310d1689/volumes"
Jul 17 22:04:36.166508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:36.166468    2546 generic.go:356] "Generic (PLEG): container finished" podID="b5f0e76e-e33a-4fa0-82d9-af195318c546" containerID="838448d3295eedcf19df7f95671a0165dc73636acbe25ef0b716476b5bffd763" exitCode=1
Jul 17 22:04:36.166998 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:36.166539    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2" event={"ID":"b5f0e76e-e33a-4fa0-82d9-af195318c546","Type":"ContainerDied","Data":"838448d3295eedcf19df7f95671a0165dc73636acbe25ef0b716476b5bffd763"}
Jul 17 22:04:36.166998 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:36.166919    2546 scope.go:117] "RemoveContainer" containerID="838448d3295eedcf19df7f95671a0165dc73636acbe25ef0b716476b5bffd763"
Jul 17 22:04:37.173884 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:37.173842    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2" event={"ID":"b5f0e76e-e33a-4fa0-82d9-af195318c546","Type":"ContainerStarted","Data":"bfee4486689b53bd87235bf5b84efcf0ed37f56b72fe3d0cf2501b19dae2a7ce"}
Jul 17 22:04:37.174290 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:37.174085    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 22:04:37.530839 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:37.530810    2546 scope.go:117] "RemoveContainer" containerID="cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c"
Jul 17 22:04:38.180857 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:38.180830    2546 generic.go:356] "Generic (PLEG): container finished" podID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerID="d39fd854d9c04c19d3bea4dd887e5520f2cbf9c14e15605bf8c50606d32a312e" exitCode=6
Jul 17 22:04:38.181173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:38.180896    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" event={"ID":"ed3f268b-f579-43a3-8c24-38a362b229fd","Type":"ContainerDied","Data":"d39fd854d9c04c19d3bea4dd887e5520f2cbf9c14e15605bf8c50606d32a312e"}
Jul 17 22:04:38.181173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:38.180950    2546 scope.go:117] "RemoveContainer" containerID="cbe5275bafb2d1fc6c746a065f418a0b9ad9d6aeaef4e82034666d626c8d558c"
Jul 17 22:04:38.181331 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:38.181317    2546 scope.go:117] "RemoveContainer" containerID="d39fd854d9c04c19d3bea4dd887e5520f2cbf9c14e15605bf8c50606d32a312e"
Jul 17 22:04:38.181520 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:38.181501    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr_odh-ai-gateway-infra(ed3f268b-f579-43a3-8c24-38a362b229fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd"
Jul 17 22:04:39.209837 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:39.209804    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:39.341425 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:39.341397    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:39.483226 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:39.483149    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca\") pod \"ed3f268b-f579-43a3-8c24-38a362b229fd\" (UID: \"ed3f268b-f579-43a3-8c24-38a362b229fd\") "
Jul 17 22:04:39.483514 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:39.483491    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ed3f268b-f579-43a3-8c24-38a362b229fd" (UID: "ed3f268b-f579-43a3-8c24-38a362b229fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:39.583653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:39.583620    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ed3f268b-f579-43a3-8c24-38a362b229fd-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:04:40.196291 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.196260    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"
Jul 17 22:04:40.196484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.196299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr" event={"ID":"ed3f268b-f579-43a3-8c24-38a362b229fd","Type":"ContainerDied","Data":"131fb824758db81affad45120291048e8679a9dd19bb4f84e9be52c28d8b67a0"}
Jul 17 22:04:40.196484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.196363    2546 scope.go:117] "RemoveContainer" containerID="d39fd854d9c04c19d3bea4dd887e5520f2cbf9c14e15605bf8c50606d32a312e"
Jul 17 22:04:40.217773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.217753    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:40.223666 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.223642    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-phmvr"]
Jul 17 22:04:40.536073 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.536048    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" path="/var/lib/kubelet/pods/ed3f268b-f579-43a3-8c24-38a362b229fd/volumes"
Jul 17 22:04:40.653659 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.653625    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"]
Jul 17 22:04:40.653988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.653975    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.653992    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654010    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654017    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654024    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654029    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654037    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654044 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654042    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654108    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.654254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654118    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654124    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654130    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:40.654254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.654139    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:40.658390 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.658371    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:40.664531 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.664510    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"]
Jul 17 22:04:40.794682 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.794613    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp\" (UID: \"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:40.895521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.895497    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp\" (UID: \"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:40.896166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.896148    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp\" (UID: \"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:40.971003 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:40.970979    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:41.102069 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:41.102039    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"]
Jul 17 22:04:41.103109 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:04:41.103083    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6a9f1fb_b6ab_4dc6_9b92_c9b4370407ed.slice/crio-8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366 WatchSource:0}: Error finding container 8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366: Status 404 returned error can't find the container with id 8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366
Jul 17 22:04:41.202947 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:41.202920    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp" event={"ID":"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed","Type":"ContainerStarted","Data":"8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366"}
Jul 17 22:04:41.530814 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:41.530790    2546 scope.go:117] "RemoveContainer" containerID="351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214"
Jul 17 22:04:42.210949 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.210912    2546 generic.go:356] "Generic (PLEG): container finished" podID="b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed" containerID="2fa899350fc5c5d10292dcb26dc2f7938e3476179c627519ab71e2b0d6c962e1" exitCode=0
Jul 17 22:04:42.211068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.210976    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp" event={"ID":"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed","Type":"ContainerDied","Data":"2fa899350fc5c5d10292dcb26dc2f7938e3476179c627519ab71e2b0d6c962e1"}
Jul 17 22:04:42.212768 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.212746    2546 generic.go:356] "Generic (PLEG): container finished" podID="ac5509b9-b75a-4ca6-8095-21051af90514" containerID="0f43e750f1d8d1f25ab5cf807dd55944591a3c9eed00967ee1601a5206cade3b" exitCode=6
Jul 17 22:04:42.212880 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.212777    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" event={"ID":"ac5509b9-b75a-4ca6-8095-21051af90514","Type":"ContainerDied","Data":"0f43e750f1d8d1f25ab5cf807dd55944591a3c9eed00967ee1601a5206cade3b"}
Jul 17 22:04:42.212880 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.212827    2546 scope.go:117] "RemoveContainer" containerID="351c2dac24a46a9e725a3b968cbf111e681cf151c75834449e4022cde8edb214"
Jul 17 22:04:42.213091 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:42.213077    2546 scope.go:117] "RemoveContainer" containerID="0f43e750f1d8d1f25ab5cf807dd55944591a3c9eed00967ee1601a5206cade3b"
Jul 17 22:04:42.213334 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:42.213307    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw_odh-ai-gateway-infra(ac5509b9-b75a-4ca6-8095-21051af90514)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" podUID="ac5509b9-b75a-4ca6-8095-21051af90514"
Jul 17 22:04:43.256341 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.256310    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:43.355186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.355162    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:43.396069 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.396050    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:43.517691 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.517670    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca\") pod \"ac5509b9-b75a-4ca6-8095-21051af90514\" (UID: \"ac5509b9-b75a-4ca6-8095-21051af90514\") "
Jul 17 22:04:43.517863 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.517704    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca\") pod \"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed\" (UID: \"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed\") "
Jul 17 22:04:43.518088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.518059    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac5509b9-b75a-4ca6-8095-21051af90514" (UID: "ac5509b9-b75a-4ca6-8095-21051af90514"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:43.518088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.518074    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed" (UID: "b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:04:43.619191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.619168    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac5509b9-b75a-4ca6-8095-21051af90514-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:04:43.619266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:43.619192    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:04:44.227006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.226968    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp"
Jul 17 22:04:44.227166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.227025    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp" event={"ID":"b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed","Type":"ContainerDied","Data":"8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366"}
Jul 17 22:04:44.227166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.227077    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8591088a70b122f4064b3d786c1c11606344ba70a04600e0a2762edbf7508366"
Jul 17 22:04:44.228418 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.228395    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw" event={"ID":"ac5509b9-b75a-4ca6-8095-21051af90514","Type":"ContainerDied","Data":"b2de0e5289f01ac684d2c6480e7e57bfea65d345f201004be048f64777dff2a8"}
Jul 17 22:04:44.228418 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.228418    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"
Jul 17 22:04:44.228566 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.228447    2546 scope.go:117] "RemoveContainer" containerID="0f43e750f1d8d1f25ab5cf807dd55944591a3c9eed00967ee1601a5206cade3b"
Jul 17 22:04:44.255495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.255465    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:44.259561 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.259531    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-q9kfw"]
Jul 17 22:04:44.536812 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:44.536787    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" path="/var/lib/kubelet/pods/ac5509b9-b75a-4ca6-8095-21051af90514/volumes"
Jul 17 22:04:48.115681 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115645    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115968    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115979    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115987    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115992    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.115999    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116004    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116020    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116026    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116031    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116036    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="dcde46a8-ee85-45b0-a9d0-3a1a310d1689" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116043    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed" containerName="revoke-keys"
Jul 17 22:04:48.116065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116048    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed" containerName="revoke-keys"
Jul 17 22:04:48.116437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116104    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.116437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116111    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed3f268b-f579-43a3-8c24-38a362b229fd" containerName="revoke-keys"
Jul 17 22:04:48.116437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116118    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed" containerName="revoke-keys"
Jul 17 22:04:48.116437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.116124    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:04:48.119972 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.119953    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:04:48.122359 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.122337    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:04:48.125550 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.125521    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:04:48.182498 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.182479    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-76f7bc5b6f-qd4k2"
Jul 17 22:04:48.254133 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.254102    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn\" (UID: \"fe436cd4-768d-44b0-9bad-94b40b7c694d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:04:48.355022 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.354987    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn\" (UID: \"fe436cd4-768d-44b0-9bad-94b40b7c694d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:04:48.355655 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.355635    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn\" (UID: \"fe436cd4-768d-44b0-9bad-94b40b7c694d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:04:48.431886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.431833    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:04:48.560989 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:48.560961    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:04:48.563739 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:04:48.563701    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe436cd4_768d_44b0_9bad_94b40b7c694d.slice/crio-0b763f4aaef6db67bed0218169c8ee4544213063b89f77738e43c52a4346ae53 WatchSource:0}: Error finding container 0b763f4aaef6db67bed0218169c8ee4544213063b89f77738e43c52a4346ae53: Status 404 returned error can't find the container with id 0b763f4aaef6db67bed0218169c8ee4544213063b89f77738e43c52a4346ae53
Jul 17 22:04:49.263775 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:49.263745    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerID="c3b5b8a065d744b4937a3a6bff6989bb1f48e1aae71697f6edac7d206f89607d" exitCode=6
Jul 17 22:04:49.264056 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:49.263834    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" event={"ID":"fe436cd4-768d-44b0-9bad-94b40b7c694d","Type":"ContainerDied","Data":"c3b5b8a065d744b4937a3a6bff6989bb1f48e1aae71697f6edac7d206f89607d"}
Jul 17 22:04:49.264056 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:49.263889    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" event={"ID":"fe436cd4-768d-44b0-9bad-94b40b7c694d","Type":"ContainerStarted","Data":"0b763f4aaef6db67bed0218169c8ee4544213063b89f77738e43c52a4346ae53"}
Jul 17 22:04:49.264056 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:49.263952    2546 scope.go:117] "RemoveContainer" containerID="c3b5b8a065d744b4937a3a6bff6989bb1f48e1aae71697f6edac7d206f89607d"
Jul 17 22:04:50.270155 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:50.270126    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerID="ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d" exitCode=6
Jul 17 22:04:50.270575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:50.270166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" event={"ID":"fe436cd4-768d-44b0-9bad-94b40b7c694d","Type":"ContainerDied","Data":"ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d"}
Jul 17 22:04:50.270575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:50.270219    2546 scope.go:117] "RemoveContainer" containerID="c3b5b8a065d744b4937a3a6bff6989bb1f48e1aae71697f6edac7d206f89607d"
Jul 17 22:04:50.270575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:50.270372    2546 scope.go:117] "RemoveContainer" containerID="ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d"
Jul 17 22:04:50.270742 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:50.270575    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn_odh-ai-gateway-infra(fe436cd4-768d-44b0-9bad-94b40b7c694d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d"
Jul 17 22:04:51.276963 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:04:51.276930    2546 scope.go:117] "RemoveContainer" containerID="ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d"
Jul 17 22:04:51.277361 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:04:51.277133    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn_odh-ai-gateway-infra(fe436cd4-768d-44b0-9bad-94b40b7c694d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d"
Jul 17 22:05:03.531394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:03.531313    2546 scope.go:117] "RemoveContainer" containerID="ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d"
Jul 17 22:05:04.350300 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:04.350264    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerID="2ad0a8bf3eeba79c77e9629cf654c4b761b201e78541743d299d51f29c126d19" exitCode=6
Jul 17 22:05:04.350429 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:04.350336    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" event={"ID":"fe436cd4-768d-44b0-9bad-94b40b7c694d","Type":"ContainerDied","Data":"2ad0a8bf3eeba79c77e9629cf654c4b761b201e78541743d299d51f29c126d19"}
Jul 17 22:05:04.350429 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:04.350424    2546 scope.go:117] "RemoveContainer" containerID="ef72554cb75c9440b8bfd30117de5b4c662e228873260d8494845c0faaf0311d"
Jul 17 22:05:04.350781 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:04.350753    2546 scope.go:117] "RemoveContainer" containerID="2ad0a8bf3eeba79c77e9629cf654c4b761b201e78541743d299d51f29c126d19"
Jul 17 22:05:04.351021 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:04.350993    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn_odh-ai-gateway-infra(fe436cd4-768d-44b0-9bad-94b40b7c694d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d"
Jul 17 22:05:05.377040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:05.376973    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:05:05.508531 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:05.508508    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:05:05.575302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:05.575278    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca\") pod \"fe436cd4-768d-44b0-9bad-94b40b7c694d\" (UID: \"fe436cd4-768d-44b0-9bad-94b40b7c694d\") "
Jul 17 22:05:05.575589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:05.575567    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe436cd4-768d-44b0-9bad-94b40b7c694d" (UID: "fe436cd4-768d-44b0-9bad-94b40b7c694d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:05.676341 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:05.676289    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe436cd4-768d-44b0-9bad-94b40b7c694d-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:06.362588 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.362554    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn" event={"ID":"fe436cd4-768d-44b0-9bad-94b40b7c694d","Type":"ContainerDied","Data":"0b763f4aaef6db67bed0218169c8ee4544213063b89f77738e43c52a4346ae53"}
Jul 17 22:05:06.362786 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.362615    2546 scope.go:117] "RemoveContainer" containerID="2ad0a8bf3eeba79c77e9629cf654c4b761b201e78541743d299d51f29c126d19"
Jul 17 22:05:06.362786 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.362565    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"
Jul 17 22:05:06.383708 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.383684    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:05:06.389043 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.389022    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-4tvmn"]
Jul 17 22:05:06.542647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:06.542615    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" path="/var/lib/kubelet/pods/fe436cd4-768d-44b0-9bad-94b40b7c694d/volumes"
Jul 17 22:05:09.046378 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046342    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:09.046933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046812    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.046933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046833    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.046933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046848    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.046933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046857    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.047154 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046973    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.047154 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046985    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:09.047154 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.046996    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5509b9-b75a-4ca6-8095-21051af90514" containerName="revoke-keys"
Jul 17 22:05:09.051539 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.051515    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:09.053885 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.053860    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:05:09.058455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.058429    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:09.098037 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.098004    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th\" (UID: \"612d42ae-cc00-4dca-98a1-b6effaa3f4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:09.199237 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.199208    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th\" (UID: \"612d42ae-cc00-4dca-98a1-b6effaa3f4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:09.199931 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.199909    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th\" (UID: \"612d42ae-cc00-4dca-98a1-b6effaa3f4bc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:09.364900 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.364829    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:09.496695 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:09.496670    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:09.498661 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:09.498635    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod612d42ae_cc00_4dca_98a1_b6effaa3f4bc.slice/crio-cb9f09fc218a11500c7ab062345e2c1d059756d88a5aae8f9d1b2daec9335638 WatchSource:0}: Error finding container cb9f09fc218a11500c7ab062345e2c1d059756d88a5aae8f9d1b2daec9335638: Status 404 returned error can't find the container with id cb9f09fc218a11500c7ab062345e2c1d059756d88a5aae8f9d1b2daec9335638
Jul 17 22:05:10.387548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.387451    2546 generic.go:356] "Generic (PLEG): container finished" podID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerID="f0449d2d9288097f5491ecaa000b72508cdb08bdbd14f80641ecf610924c706e" exitCode=6
Jul 17 22:05:10.387970 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.387544    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" event={"ID":"612d42ae-cc00-4dca-98a1-b6effaa3f4bc","Type":"ContainerDied","Data":"f0449d2d9288097f5491ecaa000b72508cdb08bdbd14f80641ecf610924c706e"}
Jul 17 22:05:10.387970 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.387615    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" event={"ID":"612d42ae-cc00-4dca-98a1-b6effaa3f4bc","Type":"ContainerStarted","Data":"cb9f09fc218a11500c7ab062345e2c1d059756d88a5aae8f9d1b2daec9335638"}
Jul 17 22:05:10.387970 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.387744    2546 scope.go:117] "RemoveContainer" containerID="f0449d2d9288097f5491ecaa000b72508cdb08bdbd14f80641ecf610924c706e"
Jul 17 22:05:10.879924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.879896    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:10.880249 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.880234    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:10.880249 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.880250    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:10.880403 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.880316    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe436cd4-768d-44b0-9bad-94b40b7c694d" containerName="revoke-keys"
Jul 17 22:05:10.883158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.883142    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:10.891125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.891100    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:10.911502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:10.911481    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd\" (UID: \"83bb42de-69a2-4730-af2d-99c72c7b4744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:11.012639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.012610    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd\" (UID: \"83bb42de-69a2-4730-af2d-99c72c7b4744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:11.013175 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.013156    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd\" (UID: \"83bb42de-69a2-4730-af2d-99c72c7b4744\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:11.194517 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.194493    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:11.323447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.323424    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:11.325669 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:11.325648    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83bb42de_69a2_4730_af2d_99c72c7b4744.slice/crio-b256a98850ba3e36b8bda573132ab344a315d4e19239f20e77837ed36b45db7d WatchSource:0}: Error finding container b256a98850ba3e36b8bda573132ab344a315d4e19239f20e77837ed36b45db7d: Status 404 returned error can't find the container with id b256a98850ba3e36b8bda573132ab344a315d4e19239f20e77837ed36b45db7d
Jul 17 22:05:11.396792 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.396765    2546 generic.go:356] "Generic (PLEG): container finished" podID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerID="b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f" exitCode=6
Jul 17 22:05:11.397183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.396851    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" event={"ID":"612d42ae-cc00-4dca-98a1-b6effaa3f4bc","Type":"ContainerDied","Data":"b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f"}
Jul 17 22:05:11.397183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.396917    2546 scope.go:117] "RemoveContainer" containerID="f0449d2d9288097f5491ecaa000b72508cdb08bdbd14f80641ecf610924c706e"
Jul 17 22:05:11.397183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.397020    2546 scope.go:117] "RemoveContainer" containerID="b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f"
Jul 17 22:05:11.397349 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:11.397224    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th_odh-ai-gateway-infra(612d42ae-cc00-4dca-98a1-b6effaa3f4bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc"
Jul 17 22:05:11.398183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:11.398114    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" event={"ID":"83bb42de-69a2-4730-af2d-99c72c7b4744","Type":"ContainerStarted","Data":"b256a98850ba3e36b8bda573132ab344a315d4e19239f20e77837ed36b45db7d"}
Jul 17 22:05:12.404220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:12.404185    2546 generic.go:356] "Generic (PLEG): container finished" podID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerID="73e22e60dbd0a447f21a5cb600ce5c8e0a6c5990ff2e48231e84bbe50fda5eb1" exitCode=6
Jul 17 22:05:12.404688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:12.404276    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" event={"ID":"83bb42de-69a2-4730-af2d-99c72c7b4744","Type":"ContainerDied","Data":"73e22e60dbd0a447f21a5cb600ce5c8e0a6c5990ff2e48231e84bbe50fda5eb1"}
Jul 17 22:05:12.404688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:12.404436    2546 scope.go:117] "RemoveContainer" containerID="73e22e60dbd0a447f21a5cb600ce5c8e0a6c5990ff2e48231e84bbe50fda5eb1"
Jul 17 22:05:12.406210 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:12.406193    2546 scope.go:117] "RemoveContainer" containerID="b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f"
Jul 17 22:05:12.406360 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:12.406342    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th_odh-ai-gateway-infra(612d42ae-cc00-4dca-98a1-b6effaa3f4bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc"
Jul 17 22:05:13.413891 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:13.413857    2546 generic.go:356] "Generic (PLEG): container finished" podID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerID="cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186" exitCode=6
Jul 17 22:05:13.414321 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:13.413919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" event={"ID":"83bb42de-69a2-4730-af2d-99c72c7b4744","Type":"ContainerDied","Data":"cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186"}
Jul 17 22:05:13.414321 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:13.413971    2546 scope.go:117] "RemoveContainer" containerID="73e22e60dbd0a447f21a5cb600ce5c8e0a6c5990ff2e48231e84bbe50fda5eb1"
Jul 17 22:05:13.414321 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:13.414165    2546 scope.go:117] "RemoveContainer" containerID="cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186"
Jul 17 22:05:13.414447 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:13.414393    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd_odh-ai-gateway-infra(83bb42de-69a2-4730-af2d-99c72c7b4744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744"
Jul 17 22:05:14.420714 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:14.420683    2546 scope.go:117] "RemoveContainer" containerID="cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186"
Jul 17 22:05:14.421149 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:14.420873    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd_odh-ai-gateway-infra(83bb42de-69a2-4730-af2d-99c72c7b4744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744"
Jul 17 22:05:15.635594 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.635559    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:15.640022 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.640006    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:15.645376 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.645348    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z\" (UID: \"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:15.647629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.647488    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:15.745998 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.745971    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z\" (UID: \"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:15.746508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.746490    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z\" (UID: \"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:15.952094 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:15.952030    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:16.079179 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.079156    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:16.081094 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:16.081067    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f3b6c87_c8ed_4c6c_a560_6b698dc08d8c.slice/crio-cb1d7d077f49ced363b370a2fcc665fb06900515df4f22027fd8eccc935a6ab9 WatchSource:0}: Error finding container cb1d7d077f49ced363b370a2fcc665fb06900515df4f22027fd8eccc935a6ab9: Status 404 returned error can't find the container with id cb1d7d077f49ced363b370a2fcc665fb06900515df4f22027fd8eccc935a6ab9
Jul 17 22:05:16.374615 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.374590    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:05:16.374828 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.374804    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy" containerID="cri-o://5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa" gracePeriod=30
Jul 17 22:05:16.432189 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.432162    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerStarted","Data":"5ea5a13c8a8946157d17bba7f3fde155490b405d9a71572f0b37776a36d40535"}
Jul 17 22:05:16.432323 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.432206    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerStarted","Data":"cb1d7d077f49ced363b370a2fcc665fb06900515df4f22027fd8eccc935a6ab9"}
Jul 17 22:05:16.446881 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:16.446595    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" podStartSLOduration=1.446580587 podStartE2EDuration="1.446580587s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:16.446514891 +0000 UTC m=+2892.426885369" watchObservedRunningTime="2026-07-17 22:05:16.446580587 +0000 UTC m=+2892.426951066"
Jul 17 22:05:17.439426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:17.439386    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerID="5ea5a13c8a8946157d17bba7f3fde155490b405d9a71572f0b37776a36d40535" exitCode=6
Jul 17 22:05:17.440006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:17.439470    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerDied","Data":"5ea5a13c8a8946157d17bba7f3fde155490b405d9a71572f0b37776a36d40535"}
Jul 17 22:05:17.440006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:17.439640    2546 scope.go:117] "RemoveContainer" containerID="5ea5a13c8a8946157d17bba7f3fde155490b405d9a71572f0b37776a36d40535"
Jul 17 22:05:18.447176 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:18.447083    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerID="bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a" exitCode=6
Jul 17 22:05:18.447636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:18.447182    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerDied","Data":"bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a"}
Jul 17 22:05:18.447636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:18.447261    2546 scope.go:117] "RemoveContainer" containerID="5ea5a13c8a8946157d17bba7f3fde155490b405d9a71572f0b37776a36d40535"
Jul 17 22:05:18.447636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:18.447329    2546 scope.go:117] "RemoveContainer" containerID="bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a"
Jul 17 22:05:18.447636 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:18.447531    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z_odh-ai-gateway-infra(2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c"
Jul 17 22:05:19.458943 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:19.458915    2546 scope.go:117] "RemoveContainer" containerID="bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a"
Jul 17 22:05:19.459375 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:19.459110    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z_odh-ai-gateway-infra(2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c"
Jul 17 22:05:21.635988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.635965    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:05:21.694877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.694814    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.694877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.694849    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695059 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.694877    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695059 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.694900    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7dhhh\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695059 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.694953    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695072    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:21.695224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695126    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695164    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695184    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:21.695224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695199    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695252    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data\") pod \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\" (UID: \"9c9c912a-5de1-4201-9eba-57cd5a10ca81\") "
Jul 17 22:05:21.695565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695516    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.695565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695520    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:21.695565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695536    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.695927 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.695860    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:21.696349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.696317    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data" (OuterVolumeSpecName: "istio-data") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:21.697302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.697272    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:05:21.697302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.697272    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:21.697456 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.697314    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh" (OuterVolumeSpecName: "kube-api-access-7dhhh") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "kube-api-access-7dhhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:05:21.697605 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.697591    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token" (OuterVolumeSpecName: "istio-token") pod "9c9c912a-5de1-4201-9eba-57cd5a10ca81" (UID: "9c9c912a-5de1-4201-9eba-57cd5a10ca81"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:05:21.796001 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.795977    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796001 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.795996    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dhhh\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-kube-api-access-7dhhh\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.796005    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.796014    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.796022    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.796029    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:21.796131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:21.796037    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c9c912a-5de1-4201-9eba-57cd5a10ca81-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:22.476962 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.476922    2546 generic.go:356] "Generic (PLEG): container finished" podID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerID="5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa" exitCode=0
Jul 17 22:05:22.477148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.476989    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"
Jul 17 22:05:22.477148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.477005    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerDied","Data":"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"}
Jul 17 22:05:22.477148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.477090    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv" event={"ID":"9c9c912a-5de1-4201-9eba-57cd5a10ca81","Type":"ContainerDied","Data":"fa104fec618cd5a1d90f49ccfcf1e3fd576e0fc76c6de51165e93e46f8bd02a0"}
Jul 17 22:05:22.477148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.477113    2546 scope.go:117] "RemoveContainer" containerID="5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"
Jul 17 22:05:22.486940 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.486924    2546 scope.go:117] "RemoveContainer" containerID="d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"
Jul 17 22:05:22.495434 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.495259    2546 scope.go:117] "RemoveContainer" containerID="5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"
Jul 17 22:05:22.495916 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:22.495886    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa\": container with ID starting with 5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa not found: ID does not exist" containerID="5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"
Jul 17 22:05:22.496014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.495925    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa"} err="failed to get container status \"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa\": rpc error: code = NotFound desc = could not find container \"5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa\": container with ID starting with 5275d5e16ff4e160459c5d09b9cb47e170ad2a8520d056ca422520f9bee481fa not found: ID does not exist"
Jul 17 22:05:22.496014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.495943    2546 scope.go:117] "RemoveContainer" containerID="d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"
Jul 17 22:05:22.496196 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:22.496178    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54\": container with ID starting with d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54 not found: ID does not exist" containerID="d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"
Jul 17 22:05:22.496248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.496202    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54"} err="failed to get container status \"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54\": rpc error: code = NotFound desc = could not find container \"d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54\": container with ID starting with d54c1bdd4924922087a3772ddfb90023bdeba4c380c466e4bf3effe8dae32a54 not found: ID does not exist"
Jul 17 22:05:22.498381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.498359    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:05:22.503676 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.503656    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-592f58-openshift-default-644c64bb67-w74dv"]
Jul 17 22:05:22.536703 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:22.536677    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" path="/var/lib/kubelet/pods/9c9c912a-5de1-4201-9eba-57cd5a10ca81/volumes"
Jul 17 22:05:24.533571 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:24.533537    2546 scope.go:117] "RemoveContainer" containerID="b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f"
Jul 17 22:05:25.496262 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:25.496234    2546 generic.go:356] "Generic (PLEG): container finished" podID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerID="516e961e9dcd9bdd92d23741361fb25a6fa4171cbf2cedce54bada972eb3a176" exitCode=6
Jul 17 22:05:25.496446 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:25.496312    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" event={"ID":"612d42ae-cc00-4dca-98a1-b6effaa3f4bc","Type":"ContainerDied","Data":"516e961e9dcd9bdd92d23741361fb25a6fa4171cbf2cedce54bada972eb3a176"}
Jul 17 22:05:25.496446 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:25.496374    2546 scope.go:117] "RemoveContainer" containerID="b07d1564ab2b0d63595444ff9b6fe961b5affd03d364030c0f2a8af1c4607a6f"
Jul 17 22:05:25.496736 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:25.496704    2546 scope.go:117] "RemoveContainer" containerID="516e961e9dcd9bdd92d23741361fb25a6fa4171cbf2cedce54bada972eb3a176"
Jul 17 22:05:25.496953 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:25.496929    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th_odh-ai-gateway-infra(612d42ae-cc00-4dca-98a1-b6effaa3f4bc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc"
Jul 17 22:05:26.523879 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.523850    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:26.531279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.531258    2546 scope.go:117] "RemoveContainer" containerID="cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186"
Jul 17 22:05:26.660785 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.660765    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:26.732158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.732132    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca\") pod \"612d42ae-cc00-4dca-98a1-b6effaa3f4bc\" (UID: \"612d42ae-cc00-4dca-98a1-b6effaa3f4bc\") "
Jul 17 22:05:26.732487 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.732463    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "612d42ae-cc00-4dca-98a1-b6effaa3f4bc" (UID: "612d42ae-cc00-4dca-98a1-b6effaa3f4bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:26.740262 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740239    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"]
Jul 17 22:05:26.740556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740545    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740558    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740567    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740572    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740578    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740585    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740598    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740604    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740611    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740616    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740659    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740667    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740673    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.740827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.740678    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" containerName="revoke-keys"
Jul 17 22:05:26.743559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.743546    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:26.751440 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.751417    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"]
Jul 17 22:05:26.832698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.832642    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8\" (UID: \"711b4a98-240f-43bc-ad9e-25909b9c4d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:26.832817 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.832743    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/612d42ae-cc00-4dca-98a1-b6effaa3f4bc-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:26.933147 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.933126    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8\" (UID: \"711b4a98-240f-43bc-ad9e-25909b9c4d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:26.933693 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:26.933676    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8\" (UID: \"711b4a98-240f-43bc-ad9e-25909b9c4d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:27.054834 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.054812    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:27.189605 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.189579    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"]
Jul 17 22:05:27.220502 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:27.220477    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod711b4a98_240f_43bc_ad9e_25909b9c4d6c.slice/crio-46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab WatchSource:0}: Error finding container 46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab: Status 404 returned error can't find the container with id 46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab
Jul 17 22:05:27.510021 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.509992    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"
Jul 17 22:05:27.510213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.510036    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th" event={"ID":"612d42ae-cc00-4dca-98a1-b6effaa3f4bc","Type":"ContainerDied","Data":"cb9f09fc218a11500c7ab062345e2c1d059756d88a5aae8f9d1b2daec9335638"}
Jul 17 22:05:27.510213 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.510096    2546 scope.go:117] "RemoveContainer" containerID="516e961e9dcd9bdd92d23741361fb25a6fa4171cbf2cedce54bada972eb3a176"
Jul 17 22:05:27.512150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.512123    2546 generic.go:356] "Generic (PLEG): container finished" podID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerID="6d9f0c8fd6d043b1e571db19c06ce5a95b9c3d0b4dc1cdbdda7a21c233d891d5" exitCode=6
Jul 17 22:05:27.512287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.512177    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" event={"ID":"83bb42de-69a2-4730-af2d-99c72c7b4744","Type":"ContainerDied","Data":"6d9f0c8fd6d043b1e571db19c06ce5a95b9c3d0b4dc1cdbdda7a21c233d891d5"}
Jul 17 22:05:27.512535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.512518    2546 scope.go:117] "RemoveContainer" containerID="6d9f0c8fd6d043b1e571db19c06ce5a95b9c3d0b4dc1cdbdda7a21c233d891d5"
Jul 17 22:05:27.512794 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:27.512769    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd_odh-ai-gateway-infra(83bb42de-69a2-4730-af2d-99c72c7b4744)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744"
Jul 17 22:05:27.513919 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.513893    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8" event={"ID":"711b4a98-240f-43bc-ad9e-25909b9c4d6c","Type":"ContainerStarted","Data":"ed7062d7038dca60d8fd3e002589b0356a3eed18bd39dfd1bbadf7ceaada9479"}
Jul 17 22:05:27.514022 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.513949    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8" event={"ID":"711b4a98-240f-43bc-ad9e-25909b9c4d6c","Type":"ContainerStarted","Data":"46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab"}
Jul 17 22:05:27.520041 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.520025    2546 scope.go:117] "RemoveContainer" containerID="cf832d66924be57d84125b7b34a8641a23044a9b9386cf96b2b31bcb0f069186"
Jul 17 22:05:27.541528 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.541488    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8" podStartSLOduration=1.5414767820000002 podStartE2EDuration="1.541476782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:27.540498357 +0000 UTC m=+2903.520868834" watchObservedRunningTime="2026-07-17 22:05:27.541476782 +0000 UTC m=+2903.521847259"
Jul 17 22:05:27.574747 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.573588    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:27.580984 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:27.580952    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-kr4th"]
Jul 17 22:05:28.524157 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.524118    2546 generic.go:356] "Generic (PLEG): container finished" podID="711b4a98-240f-43bc-ad9e-25909b9c4d6c" containerID="ed7062d7038dca60d8fd3e002589b0356a3eed18bd39dfd1bbadf7ceaada9479" exitCode=0
Jul 17 22:05:28.524364 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.524167    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8" event={"ID":"711b4a98-240f-43bc-ad9e-25909b9c4d6c","Type":"ContainerDied","Data":"ed7062d7038dca60d8fd3e002589b0356a3eed18bd39dfd1bbadf7ceaada9479"}
Jul 17 22:05:28.537601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.537573    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="612d42ae-cc00-4dca-98a1-b6effaa3f4bc" path="/var/lib/kubelet/pods/612d42ae-cc00-4dca-98a1-b6effaa3f4bc/volumes"
Jul 17 22:05:28.568311 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.568283    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:28.699762 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.699739    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:28.749279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.749251    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca\") pod \"83bb42de-69a2-4730-af2d-99c72c7b4744\" (UID: \"83bb42de-69a2-4730-af2d-99c72c7b4744\") "
Jul 17 22:05:28.749548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.749528    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83bb42de-69a2-4730-af2d-99c72c7b4744" (UID: "83bb42de-69a2-4730-af2d-99c72c7b4744"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:28.850688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:28.850632    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83bb42de-69a2-4730-af2d-99c72c7b4744-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:29.531070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.531044    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"
Jul 17 22:05:29.531267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.531082    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd" event={"ID":"83bb42de-69a2-4730-af2d-99c72c7b4744","Type":"ContainerDied","Data":"b256a98850ba3e36b8bda573132ab344a315d4e19239f20e77837ed36b45db7d"}
Jul 17 22:05:29.531267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.531122    2546 scope.go:117] "RemoveContainer" containerID="6d9f0c8fd6d043b1e571db19c06ce5a95b9c3d0b4dc1cdbdda7a21c233d891d5"
Jul 17 22:05:29.553151 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.553127    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:29.557111 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.557088    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-vr2rd"]
Jul 17 22:05:29.662831 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.662805    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:29.758874 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.758849    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca\") pod \"711b4a98-240f-43bc-ad9e-25909b9c4d6c\" (UID: \"711b4a98-240f-43bc-ad9e-25909b9c4d6c\") "
Jul 17 22:05:29.759162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.759140    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "711b4a98-240f-43bc-ad9e-25909b9c4d6c" (UID: "711b4a98-240f-43bc-ad9e-25909b9c4d6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:29.859574 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:29.859508    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/711b4a98-240f-43bc-ad9e-25909b9c4d6c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:30.537445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:30.537403    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" path="/var/lib/kubelet/pods/83bb42de-69a2-4730-af2d-99c72c7b4744/volumes"
Jul 17 22:05:30.537656 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:30.537617    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8"
Jul 17 22:05:30.538094 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:30.538058    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8" event={"ID":"711b4a98-240f-43bc-ad9e-25909b9c4d6c","Type":"ContainerDied","Data":"46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab"}
Jul 17 22:05:30.538206 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:30.538111    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="46bb07c72ca1b5a619cf46ffa6dd7d41d874e7a6a6d2e6a2a4dd67152fd0dfab"
Jul 17 22:05:34.534365 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:34.534326    2546 scope.go:117] "RemoveContainer" containerID="bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a"
Jul 17 22:05:35.568285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:35.568252    2546 generic.go:356] "Generic (PLEG): container finished" podID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerID="7523684e04964ef104b5666489a5a58d4736ba488fd85f9648ede9e6fd6a49db" exitCode=6
Jul 17 22:05:35.568689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:35.568329    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerDied","Data":"7523684e04964ef104b5666489a5a58d4736ba488fd85f9648ede9e6fd6a49db"}
Jul 17 22:05:35.568689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:35.568399    2546 scope.go:117] "RemoveContainer" containerID="bad1b34157f6577c3e6fe52014907f7e3503b369b51d25abe5b6d532cfd8f90a"
Jul 17 22:05:35.568689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:35.568666    2546 scope.go:117] "RemoveContainer" containerID="7523684e04964ef104b5666489a5a58d4736ba488fd85f9648ede9e6fd6a49db"
Jul 17 22:05:35.568915 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:35.568894    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z_odh-ai-gateway-infra(2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c"
Jul 17 22:05:36.594801 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:36.594762    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:36.730102 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:36.730076    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:36.821273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:36.821244    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca\") pod \"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c\" (UID: \"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c\") "
Jul 17 22:05:36.821547 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:36.821526    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" (UID: "2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:36.922392 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:36.922324    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:37.581481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:37.581449    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"
Jul 17 22:05:37.581481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:37.581464    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z" event={"ID":"2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c","Type":"ContainerDied","Data":"cb1d7d077f49ced363b370a2fcc665fb06900515df4f22027fd8eccc935a6ab9"}
Jul 17 22:05:37.581705 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:37.581523    2546 scope.go:117] "RemoveContainer" containerID="7523684e04964ef104b5666489a5a58d4736ba488fd85f9648ede9e6fd6a49db"
Jul 17 22:05:37.602827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:37.602801    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:37.607716 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:37.607694    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-5gt7z"]
Jul 17 22:05:38.157396 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157369    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:38.157751 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157711    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157755    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157768    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="711b4a98-240f-43bc-ad9e-25909b9c4d6c" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157776    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="711b4a98-240f-43bc-ad9e-25909b9c4d6c" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157790    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157798    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157822    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157830    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157842    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.157858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157850    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157864    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157872    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157961    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157973    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="711b4a98-240f-43bc-ad9e-25909b9c4d6c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157982    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.157991    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.158000    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.158009    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" containerName="revoke-keys"
Jul 17 22:05:38.158283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.158019    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c9c912a-5de1-4201-9eba-57cd5a10ca81" containerName="istio-proxy"
Jul 17 22:05:38.162497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.162478    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:38.165419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.165401    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:05:38.168048 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.168014    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:38.231858 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.231829    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt\" (UID: \"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:38.332757 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.332706    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt\" (UID: \"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:38.333332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.333310    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt\" (UID: \"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:38.473534 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.473472    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:38.537995 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.537959    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c" path="/var/lib/kubelet/pods/2f3b6c87-c8ed-4c6c-a560-6b698dc08d8c/volumes"
Jul 17 22:05:38.602175 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:38.602155    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:38.604627 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:38.604603    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a791a50_a6f6_473e_93c5_b3a7d3a3a7fe.slice/crio-93a1297152680713c9b4e12f07c686fd91b7050b862b4715d85a498eda9f8eb7 WatchSource:0}: Error finding container 93a1297152680713c9b4e12f07c686fd91b7050b862b4715d85a498eda9f8eb7: Status 404 returned error can't find the container with id 93a1297152680713c9b4e12f07c686fd91b7050b862b4715d85a498eda9f8eb7
Jul 17 22:05:39.595089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:39.595053    2546 generic.go:356] "Generic (PLEG): container finished" podID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerID="86b744357f15a2e24c18b28eab12383aaef5664d30a7b3d7700188b094b2263a" exitCode=6
Jul 17 22:05:39.595306 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:39.595091    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" event={"ID":"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe","Type":"ContainerDied","Data":"86b744357f15a2e24c18b28eab12383aaef5664d30a7b3d7700188b094b2263a"}
Jul 17 22:05:39.595306 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:39.595129    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" event={"ID":"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe","Type":"ContainerStarted","Data":"93a1297152680713c9b4e12f07c686fd91b7050b862b4715d85a498eda9f8eb7"}
Jul 17 22:05:39.595306 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:39.595265    2546 scope.go:117] "RemoveContainer" containerID="86b744357f15a2e24c18b28eab12383aaef5664d30a7b3d7700188b094b2263a"
Jul 17 22:05:40.603669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:40.603628    2546 generic.go:356] "Generic (PLEG): container finished" podID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerID="d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab" exitCode=6
Jul 17 22:05:40.604152 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:40.603739    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" event={"ID":"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe","Type":"ContainerDied","Data":"d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab"}
Jul 17 22:05:40.604152 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:40.603812    2546 scope.go:117] "RemoveContainer" containerID="86b744357f15a2e24c18b28eab12383aaef5664d30a7b3d7700188b094b2263a"
Jul 17 22:05:40.604286 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:40.604184    2546 scope.go:117] "RemoveContainer" containerID="d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab"
Jul 17 22:05:40.604421 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:40.604398    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt_odh-ai-gateway-infra(4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe"
Jul 17 22:05:41.611853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:41.611824    2546 scope.go:117] "RemoveContainer" containerID="d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab"
Jul 17 22:05:41.612249 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:41.612018    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt_odh-ai-gateway-infra(4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe"
Jul 17 22:05:52.478354 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:52.478321    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:05:52.478909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:52.478527    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy" containerID="cri-o://26f379f4b9fc7c59c79463a241ca98dfdfc2249cb261bad20b441189e666c354" gracePeriod=30
Jul 17 22:05:53.352442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.352406    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:05:53.352878 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.352863    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:53.352932 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.352882    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:53.352978 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.352970    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="83bb42de-69a2-4730-af2d-99c72c7b4744" containerName="revoke-keys"
Jul 17 22:05:53.356401 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.356380    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.359622 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.359592    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-e12150-openshift-default-dockercfg-7jc4m\""
Jul 17 22:05:53.368688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.368655    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:05:53.438400 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438351    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438400 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438388    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438423    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438517    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438562    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sthqn\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438585    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438645    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.438745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.438664    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.539610 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539568    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.539610 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539610    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539635    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539653    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539783    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sthqn\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539831    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539867    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539901    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539927    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.539995    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.540205    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.540322    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.540387    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.540551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.540540    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.542066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.542042    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.542190 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.542174    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.548320 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.548300    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sthqn\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.548445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.548425    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token\") pod \"e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") " pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.666996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.666918    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:53.803851 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.803072    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:05:53.806701 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:53.806674    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc987f5f9_9079_4794_aa22_3f13c0f4e2fa.slice/crio-6d52f40ba05b52099809d0f36b26722d3f162d02d7036f2f8fc7ed918b1db474 WatchSource:0}: Error finding container 6d52f40ba05b52099809d0f36b26722d3f162d02d7036f2f8fc7ed918b1db474: Status 404 returned error can't find the container with id 6d52f40ba05b52099809d0f36b26722d3f162d02d7036f2f8fc7ed918b1db474
Jul 17 22:05:53.808805 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.808773    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:05:53.808877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.808844    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:05:53.808877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:53.808871    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:05:54.680831 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:54.680797    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerStarted","Data":"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"}
Jul 17 22:05:54.681235 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:54.680850    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerStarted","Data":"6d52f40ba05b52099809d0f36b26722d3f162d02d7036f2f8fc7ed918b1db474"}
Jul 17 22:05:54.700146 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:54.700091    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podStartSLOduration=1.700075665 podStartE2EDuration="1.700075665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:05:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:05:54.699818902 +0000 UTC m=+2930.680189379" watchObservedRunningTime="2026-07-17 22:05:54.700075665 +0000 UTC m=+2930.680446142"
Jul 17 22:05:55.531640 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:55.531588    2546 scope.go:117] "RemoveContainer" containerID="d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab"
Jul 17 22:05:55.667321 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:55.667286    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:55.672251 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:55.672230    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:55.686647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:55.686624    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:55.687407 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:55.687391    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:05:56.694591 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:56.694559    2546 generic.go:356] "Generic (PLEG): container finished" podID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerID="d1fef3fcca302c5808676f01d22d6cbe42e67816efaec08d0367e7935af47d0f" exitCode=6
Jul 17 22:05:56.695046 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:56.694632    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" event={"ID":"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe","Type":"ContainerDied","Data":"d1fef3fcca302c5808676f01d22d6cbe42e67816efaec08d0367e7935af47d0f"}
Jul 17 22:05:56.695046 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:56.694742    2546 scope.go:117] "RemoveContainer" containerID="d15a76bf85462d8374983f113cc6b6d8622a08e853bba7d64ca26748502c74ab"
Jul 17 22:05:56.695218 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:56.695197    2546 scope.go:117] "RemoveContainer" containerID="d1fef3fcca302c5808676f01d22d6cbe42e67816efaec08d0367e7935af47d0f"
Jul 17 22:05:56.695405 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:05:56.695380    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt_odh-ai-gateway-infra(4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe"
Jul 17 22:05:57.703188 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.703161    2546 generic.go:356] "Generic (PLEG): container finished" podID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerID="26f379f4b9fc7c59c79463a241ca98dfdfc2249cb261bad20b441189e666c354" exitCode=0
Jul 17 22:05:57.703572 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.703234    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerDied","Data":"26f379f4b9fc7c59c79463a241ca98dfdfc2249cb261bad20b441189e666c354"}
Jul 17 22:05:57.703572 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.703309    2546 scope.go:117] "RemoveContainer" containerID="0c67bcc98cd781d2089ba655ab8ab8597873c0b9e6c7c8abd75d8fa779cb48d3"
Jul 17 22:05:57.735160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.735137    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:05:57.747921 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.747859    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:57.877708 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877688    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:57.877839 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877756    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.877839 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877799    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.877839 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877828    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877845    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877889    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877914    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.877961    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878011    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878058    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rvsvp\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp\") pod \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\" (UID: \"d55fc2ea-66ba-429d-8bd3-546c05fd5f67\") "
Jul 17 22:05:57.878239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878147    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:57.878239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878165    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:57.878621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878329    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:57.878621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878483    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:57.878621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878375    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.878621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878576    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.878919 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.878898    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data" (OuterVolumeSpecName: "istio-data") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:57.880017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.879994    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:05:57.880102 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.880059    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:05:57.880498 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.880478    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token" (OuterVolumeSpecName: "istio-token") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:05:57.880596 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.880494    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp" (OuterVolumeSpecName: "kube-api-access-rvsvp") pod "d55fc2ea-66ba-429d-8bd3-546c05fd5f67" (UID: "d55fc2ea-66ba-429d-8bd3-546c05fd5f67"). InnerVolumeSpecName "kube-api-access-rvsvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:05:57.979199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979171    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca\") pod \"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe\" (UID: \"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe\") "
Jul 17 22:05:57.979383 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979367    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979390    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979404    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979419    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvsvp\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-kube-api-access-rvsvp\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979433    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979446    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979460    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d55fc2ea-66ba-429d-8bd3-546c05fd5f67-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:57.979621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:57.979521    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" (UID: "4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:05:58.080181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.080118    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:05:58.709050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.709022    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"
Jul 17 22:05:58.709050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.709040    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t" event={"ID":"d55fc2ea-66ba-429d-8bd3-546c05fd5f67","Type":"ContainerDied","Data":"f8fc02f5ef4d469597307561d73191b368e9dd960e0e20eb5bc64f0d2a06f1d4"}
Jul 17 22:05:58.709518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.709136    2546 scope.go:117] "RemoveContainer" containerID="26f379f4b9fc7c59c79463a241ca98dfdfc2249cb261bad20b441189e666c354"
Jul 17 22:05:58.710520 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.710456    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt" event={"ID":"4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe","Type":"ContainerDied","Data":"93a1297152680713c9b4e12f07c686fd91b7050b862b4715d85a498eda9f8eb7"}
Jul 17 22:05:58.710520 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.710488    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"
Jul 17 22:05:58.718538 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.718519    2546 scope.go:117] "RemoveContainer" containerID="d1fef3fcca302c5808676f01d22d6cbe42e67816efaec08d0367e7935af47d0f"
Jul 17 22:05:58.729584 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.729565    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:05:58.733628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.733608    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-6be22b-openshift-default-68dc94cfbf-n865t"]
Jul 17 22:05:58.745987 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.745940    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:58.749317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:58.749299    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-8tcvt"]
Jul 17 22:05:59.094663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094593    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:05:59.094905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094893    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094908    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094917    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094922    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094933    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094939    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094951    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.094952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094955    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.095160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.094998    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.095160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.095005    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:05:59.095160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.095016    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:05:59.099527 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.099512    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:05:59.105261 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.105239    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:05:59.108700 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.108677    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:05:59.190220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.190197    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv\" (UID: \"3f463a32-7bc6-49a3-b65f-5358656350a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:05:59.290702 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.290659    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv\" (UID: \"3f463a32-7bc6-49a3-b65f-5358656350a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:05:59.291242 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.291221    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv\" (UID: \"3f463a32-7bc6-49a3-b65f-5358656350a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:05:59.409556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.409483    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:05:59.538160 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.538135    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:05:59.539963 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:05:59.539936    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f463a32_7bc6_49a3_b65f_5358656350a0.slice/crio-d68ba6be045147cf45291f84369120e4dd92aaac5e7de3d945f7298301d443d4 WatchSource:0}: Error finding container d68ba6be045147cf45291f84369120e4dd92aaac5e7de3d945f7298301d443d4: Status 404 returned error can't find the container with id d68ba6be045147cf45291f84369120e4dd92aaac5e7de3d945f7298301d443d4
Jul 17 22:05:59.718192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:05:59.718129    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" event={"ID":"3f463a32-7bc6-49a3-b65f-5358656350a0","Type":"ContainerStarted","Data":"d68ba6be045147cf45291f84369120e4dd92aaac5e7de3d945f7298301d443d4"}
Jul 17 22:06:00.537569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.537537    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" path="/var/lib/kubelet/pods/4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe/volumes"
Jul 17 22:06:00.537988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.537972    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" path="/var/lib/kubelet/pods/d55fc2ea-66ba-429d-8bd3-546c05fd5f67/volumes"
Jul 17 22:06:00.726072 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.726040    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerID="3d7af16eb7559fb1fd723eea5f2a36e1e7553c8aa31463424a329b08a853b254" exitCode=6
Jul 17 22:06:00.726509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.726092    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" event={"ID":"3f463a32-7bc6-49a3-b65f-5358656350a0","Type":"ContainerDied","Data":"3d7af16eb7559fb1fd723eea5f2a36e1e7553c8aa31463424a329b08a853b254"}
Jul 17 22:06:00.726509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.726355    2546 scope.go:117] "RemoveContainer" containerID="3d7af16eb7559fb1fd723eea5f2a36e1e7553c8aa31463424a329b08a853b254"
Jul 17 22:06:00.921853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.921821    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:00.922203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.922185    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:06:00.922293 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.922207    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:06:00.922352 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.922310    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="d55fc2ea-66ba-429d-8bd3-546c05fd5f67" containerName="istio-proxy"
Jul 17 22:06:00.922352 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.922326    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a791a50-a6f6-473e-93c5-b3a7d3a3a7fe" containerName="revoke-keys"
Jul 17 22:06:00.925191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.925172    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:00.933082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:00.932997    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:01.005798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.005764    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp\" (UID: \"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:01.106886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.106854    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp\" (UID: \"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:01.107493 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.107469    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp\" (UID: \"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:01.238011 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.237989    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:01.367449 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.367426    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:01.421781 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:01.421754    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673e2e5b_6e9f_432f_8dcc_d5c3e5e8ad7f.slice/crio-f9db84eee0fa1721965bf0bc12d3a5b92511fb5ae88457b74d952ef8042b579b WatchSource:0}: Error finding container f9db84eee0fa1721965bf0bc12d3a5b92511fb5ae88457b74d952ef8042b579b: Status 404 returned error can't find the container with id f9db84eee0fa1721965bf0bc12d3a5b92511fb5ae88457b74d952ef8042b579b
Jul 17 22:06:01.733177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.733146    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerID="fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58" exitCode=6
Jul 17 22:06:01.733530 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.733236    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" event={"ID":"3f463a32-7bc6-49a3-b65f-5358656350a0","Type":"ContainerDied","Data":"fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58"}
Jul 17 22:06:01.733530 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.733309    2546 scope.go:117] "RemoveContainer" containerID="3d7af16eb7559fb1fd723eea5f2a36e1e7553c8aa31463424a329b08a853b254"
Jul 17 22:06:01.733530 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.733492    2546 scope.go:117] "RemoveContainer" containerID="fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58"
Jul 17 22:06:01.733706 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:01.733683    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv_odh-ai-gateway-infra(3f463a32-7bc6-49a3-b65f-5358656350a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0"
Jul 17 22:06:01.735010 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.734989    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerStarted","Data":"ede5aad88f4b0caffd524c21869e506f398b99634091b61e565c65a4e4bccad2"}
Jul 17 22:06:01.735108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.735032    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerStarted","Data":"f9db84eee0fa1721965bf0bc12d3a5b92511fb5ae88457b74d952ef8042b579b"}
Jul 17 22:06:01.769119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.769079    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" podStartSLOduration=1.769066575 podStartE2EDuration="1.769066575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:01.766503925 +0000 UTC m=+2937.746874403" watchObservedRunningTime="2026-07-17 22:06:01.769066575 +0000 UTC m=+2937.749437082"
Jul 17 22:06:01.821438 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.821349    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:06:01.825953 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.825931    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.828692 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.828668    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-92c219-openshift-default-dockercfg-84wdg\""
Jul 17 22:06:01.836923 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.836902    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:06:01.914167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914141    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914183    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914214    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914289    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914326    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf7sf\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914354    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914374    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914389    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:01.914543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:01.914422    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.015931 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.015898    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.015936    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016066    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016108    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016128    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf7sf\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016159    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016185    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016357    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016424    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016436    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016612 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016503    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016612 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016518    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.016735 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.016676    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.017360 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.017328    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.018580 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.018557    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.018698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.018632    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.025133 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.025089    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.025934 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.025915    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf7sf\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf\") pod \"e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") " pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.139681 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.139582    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:02.280093 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.280071    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:06:02.281425 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:02.281395    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e8e7949_5070_4f34_bbdb_b4df824fbd23.slice/crio-38e546c89585589ac4d5d48481661bb3a236c75d32a290402f17ca8d3545a10e WatchSource:0}: Error finding container 38e546c89585589ac4d5d48481661bb3a236c75d32a290402f17ca8d3545a10e: Status 404 returned error can't find the container with id 38e546c89585589ac4d5d48481661bb3a236c75d32a290402f17ca8d3545a10e
Jul 17 22:06:02.283701 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.283673    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:02.283785 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.283760    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:02.283824 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.283798    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:02.742274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.742240    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerStarted","Data":"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"}
Jul 17 22:06:02.742705 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.742301    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerStarted","Data":"38e546c89585589ac4d5d48481661bb3a236c75d32a290402f17ca8d3545a10e"}
Jul 17 22:06:02.743944 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.743918    2546 generic.go:356] "Generic (PLEG): container finished" podID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerID="ede5aad88f4b0caffd524c21869e506f398b99634091b61e565c65a4e4bccad2" exitCode=6
Jul 17 22:06:02.744065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.744005    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerDied","Data":"ede5aad88f4b0caffd524c21869e506f398b99634091b61e565c65a4e4bccad2"}
Jul 17 22:06:02.744196 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.744162    2546 scope.go:117] "RemoveContainer" containerID="ede5aad88f4b0caffd524c21869e506f398b99634091b61e565c65a4e4bccad2"
Jul 17 22:06:02.746183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.746169    2546 scope.go:117] "RemoveContainer" containerID="fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58"
Jul 17 22:06:02.746360 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:02.746338    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv_odh-ai-gateway-infra(3f463a32-7bc6-49a3-b65f-5358656350a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0"
Jul 17 22:06:02.775455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:02.775390    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podStartSLOduration=1.775369794 podStartE2EDuration="1.775369794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:02.771364945 +0000 UTC m=+2938.751735423" watchObservedRunningTime="2026-07-17 22:06:02.775369794 +0000 UTC m=+2938.755740272"
Jul 17 22:06:03.140630 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.140536    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:03.145448 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.145422    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:03.754291 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.754265    2546 generic.go:356] "Generic (PLEG): container finished" podID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerID="b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7" exitCode=6
Jul 17 22:06:03.754615 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.754340    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerDied","Data":"b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7"}
Jul 17 22:06:03.754615 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.754415    2546 scope.go:117] "RemoveContainer" containerID="ede5aad88f4b0caffd524c21869e506f398b99634091b61e565c65a4e4bccad2"
Jul 17 22:06:03.754764 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.754666    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:03.754843 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.754814    2546 scope.go:117] "RemoveContainer" containerID="b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7"
Jul 17 22:06:03.755033 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:03.755011    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp_odh-ai-gateway-infra(673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f"
Jul 17 22:06:03.755785 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:03.755765    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:04.775301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:04.774609    2546 scope.go:117] "RemoveContainer" containerID="b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7"
Jul 17 22:06:04.775301 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:04.774808    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp_odh-ai-gateway-infra(673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f"
Jul 17 22:06:09.806264 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.806227    2546 generic.go:356] "Generic (PLEG): container finished" podID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerID="da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135" exitCode=0
Jul 17 22:06:09.806710 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.806299    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerDied","Data":"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"}
Jul 17 22:06:09.806710 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.806654    2546 scope.go:117] "RemoveContainer" containerID="da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"
Jul 17 22:06:09.807712 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.807684    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:09.807802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.807785    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:09.807851 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:09.807839    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:10.687013 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:10.686976    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:10.812854 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:10.812820    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerStarted","Data":"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"}
Jul 17 22:06:11.667445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:11.667412    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:11.668714 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:11.668681    2546 patch_prober.go:28] interesting pod/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Jul 17 22:06:11.668875 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:11.668766    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Jul 17 22:06:12.668027 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:12.667994    2546 patch_prober.go:28] interesting pod/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Jul 17 22:06:12.668589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:12.668558    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Jul 17 22:06:13.668212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:13.668173    2546 patch_prober.go:28] interesting pod/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Jul 17 22:06:13.668844 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:13.668811    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Jul 17 22:06:14.668330 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:14.668288    2546 patch_prober.go:28] interesting pod/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused" start-of-body=
Jul 17 22:06:14.669001 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:14.668965    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.131:15021/healthz/ready\": dial tcp 10.133.0.131:15021: connect: connection refused"
Jul 17 22:06:15.530811 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.530783    2546 scope.go:117] "RemoveContainer" containerID="fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58"
Jul 17 22:06:15.683239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.683200    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:15.683884 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.683860    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:15.684565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.684546    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:15.701391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.701369    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:15.704471 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.704458    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:15.717676 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.717655    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:15.846370 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.846340    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7\" (UID: \"821e299b-62da-4421-8d93-ab51a6eaf603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:15.947163 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.947132    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7\" (UID: \"821e299b-62da-4421-8d93-ab51a6eaf603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:15.947704 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:15.947685    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7\" (UID: \"821e299b-62da-4421-8d93-ab51a6eaf603\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:16.013926 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.013898    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:16.184488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.184454    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:16.188083 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:16.188057    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821e299b_62da_4421_8d93_ab51a6eaf603.slice/crio-45e57643dac5474a4af1a8b5d757f69e9b0b273fa355f8871045935c3c9c3049 WatchSource:0}: Error finding container 45e57643dac5474a4af1a8b5d757f69e9b0b273fa355f8871045935c3c9c3049: Status 404 returned error can't find the container with id 45e57643dac5474a4af1a8b5d757f69e9b0b273fa355f8871045935c3c9c3049
Jul 17 22:06:16.844815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.844787    2546 generic.go:356] "Generic (PLEG): container finished" podID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerID="146ce2a83cb1e9a7de0441100cb1d23af9298986ce2e52015b71d520944b881d" exitCode=6
Jul 17 22:06:16.845266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.844864    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" event={"ID":"3f463a32-7bc6-49a3-b65f-5358656350a0","Type":"ContainerDied","Data":"146ce2a83cb1e9a7de0441100cb1d23af9298986ce2e52015b71d520944b881d"}
Jul 17 22:06:16.845266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.844925    2546 scope.go:117] "RemoveContainer" containerID="fe4848f51c75f6ae2be215c3b7c299071ad0ff39c49dab5f60f3934cdc834d58"
Jul 17 22:06:16.845266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.845212    2546 scope.go:117] "RemoveContainer" containerID="146ce2a83cb1e9a7de0441100cb1d23af9298986ce2e52015b71d520944b881d"
Jul 17 22:06:16.845474 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:16.845451    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv_odh-ai-gateway-infra(3f463a32-7bc6-49a3-b65f-5358656350a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0"
Jul 17 22:06:16.849577 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.849555    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerStarted","Data":"5d807c652bedfc93301efac92f90a889447fecdd2d0d4920a6960de021754ef0"}
Jul 17 22:06:16.849701 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.849596    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerStarted","Data":"45e57643dac5474a4af1a8b5d757f69e9b0b273fa355f8871045935c3c9c3049"}
Jul 17 22:06:16.886379 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:16.886338    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" podStartSLOduration=1.886323318 podStartE2EDuration="1.886323318s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:16.883434679 +0000 UTC m=+2952.863805158" watchObservedRunningTime="2026-07-17 22:06:16.886323318 +0000 UTC m=+2952.866693851"
Jul 17 22:06:17.075534 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.075490    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"]
Jul 17 22:06:17.079670 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.079638    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:17.086177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.086143    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"]
Jul 17 22:06:17.167212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.167108    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e12150-95t2q\" (UID: \"771d4a38-47e9-4b4d-b59b-02dafd38e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:17.268482 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.268437    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e12150-95t2q\" (UID: \"771d4a38-47e9-4b4d-b59b-02dafd38e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:17.269328 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.269272    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-e12150-95t2q\" (UID: \"771d4a38-47e9-4b4d-b59b-02dafd38e12a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:17.453071 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.452996    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:17.531656 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.531555    2546 scope.go:117] "RemoveContainer" containerID="b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7"
Jul 17 22:06:17.643436 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.643411    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"]
Jul 17 22:06:17.645638 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:17.645609    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771d4a38_47e9_4b4d_b59b_02dafd38e12a.slice/crio-50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c WatchSource:0}: Error finding container 50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c: Status 404 returned error can't find the container with id 50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c
Jul 17 22:06:17.857588 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.857553    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q" event={"ID":"771d4a38-47e9-4b4d-b59b-02dafd38e12a","Type":"ContainerStarted","Data":"50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c"}
Jul 17 22:06:17.859495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.859458    2546 generic.go:356] "Generic (PLEG): container finished" podID="821e299b-62da-4421-8d93-ab51a6eaf603" containerID="5d807c652bedfc93301efac92f90a889447fecdd2d0d4920a6960de021754ef0" exitCode=6
Jul 17 22:06:17.859660 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.859607    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerDied","Data":"5d807c652bedfc93301efac92f90a889447fecdd2d0d4920a6960de021754ef0"}
Jul 17 22:06:17.859993 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.859970    2546 scope.go:117] "RemoveContainer" containerID="5d807c652bedfc93301efac92f90a889447fecdd2d0d4920a6960de021754ef0"
Jul 17 22:06:17.872964 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:17.872939    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:06:18.107131 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.107102    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:06:18.198550 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.198510    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca\") pod \"3f463a32-7bc6-49a3-b65f-5358656350a0\" (UID: \"3f463a32-7bc6-49a3-b65f-5358656350a0\") "
Jul 17 22:06:18.199135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.199106    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f463a32-7bc6-49a3-b65f-5358656350a0" (UID: "3f463a32-7bc6-49a3-b65f-5358656350a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:18.300100 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.300064    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f463a32-7bc6-49a3-b65f-5358656350a0-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:18.872337 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.872305    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"
Jul 17 22:06:18.872835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.872336    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv" event={"ID":"3f463a32-7bc6-49a3-b65f-5358656350a0","Type":"ContainerDied","Data":"d68ba6be045147cf45291f84369120e4dd92aaac5e7de3d945f7298301d443d4"}
Jul 17 22:06:18.872835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.872403    2546 scope.go:117] "RemoveContainer" containerID="146ce2a83cb1e9a7de0441100cb1d23af9298986ce2e52015b71d520944b881d"
Jul 17 22:06:18.874942 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.874914    2546 generic.go:356] "Generic (PLEG): container finished" podID="771d4a38-47e9-4b4d-b59b-02dafd38e12a" containerID="6fc09921cff6382d9f6e87bcc5a3403a0651267a4009014ab27ed566a2ce9f9d" exitCode=0
Jul 17 22:06:18.875061 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.875012    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q" event={"ID":"771d4a38-47e9-4b4d-b59b-02dafd38e12a","Type":"ContainerDied","Data":"6fc09921cff6382d9f6e87bcc5a3403a0651267a4009014ab27ed566a2ce9f9d"}
Jul 17 22:06:18.877359 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.877337    2546 generic.go:356] "Generic (PLEG): container finished" podID="821e299b-62da-4421-8d93-ab51a6eaf603" containerID="c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65" exitCode=6
Jul 17 22:06:18.877484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.877407    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerDied","Data":"c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65"}
Jul 17 22:06:18.877686 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.877633    2546 scope.go:117] "RemoveContainer" containerID="c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65"
Jul 17 22:06:18.877967 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:18.877879    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7_odh-ai-gateway-infra(821e299b-62da-4421-8d93-ab51a6eaf603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" podUID="821e299b-62da-4421-8d93-ab51a6eaf603"
Jul 17 22:06:18.879758 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.879711    2546 generic.go:356] "Generic (PLEG): container finished" podID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerID="b22f831b7bc7ada4e8d2289480aea4ef697c04d6b63627cf83de3554f4c46f97" exitCode=6
Jul 17 22:06:18.879830 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.879759    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerDied","Data":"b22f831b7bc7ada4e8d2289480aea4ef697c04d6b63627cf83de3554f4c46f97"}
Jul 17 22:06:18.880091 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.880077    2546 scope.go:117] "RemoveContainer" containerID="b22f831b7bc7ada4e8d2289480aea4ef697c04d6b63627cf83de3554f4c46f97"
Jul 17 22:06:18.880299 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:18.880279    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp_odh-ai-gateway-infra(673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f"
Jul 17 22:06:18.883224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.883202    2546 scope.go:117] "RemoveContainer" containerID="5d807c652bedfc93301efac92f90a889447fecdd2d0d4920a6960de021754ef0"
Jul 17 22:06:18.893632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.893612    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:06:18.895081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.895063    2546 scope.go:117] "RemoveContainer" containerID="b5018d0556f5e775e11324dbd492be9249757fc545afa0713c897889239b4ca7"
Jul 17 22:06:18.899008 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:18.898987    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-st2jv"]
Jul 17 22:06:19.796873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:19.796838    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 22:06:19.797040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:19.796938    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:06:19.887203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:19.887170    2546 scope.go:117] "RemoveContainer" containerID="c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65"
Jul 17 22:06:19.887633 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:19.887378    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7_odh-ai-gateway-infra(821e299b-62da-4421-8d93-ab51a6eaf603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" podUID="821e299b-62da-4421-8d93-ab51a6eaf603"
Jul 17 22:06:19.955330 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:19.955299    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:20.031776 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.031755    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:20.106004 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.105982    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:20.114651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.114633    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca\") pod \"771d4a38-47e9-4b4d-b59b-02dafd38e12a\" (UID: \"771d4a38-47e9-4b4d-b59b-02dafd38e12a\") "
Jul 17 22:06:20.114764 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.114742    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca\") pod \"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f\" (UID: \"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f\") "
Jul 17 22:06:20.114988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.114969    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "771d4a38-47e9-4b4d-b59b-02dafd38e12a" (UID: "771d4a38-47e9-4b4d-b59b-02dafd38e12a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:20.115067 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.115050    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" (UID: "673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:20.215905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.215875    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:20.215905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.215901    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/771d4a38-47e9-4b4d-b59b-02dafd38e12a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:20.536875 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.536836    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" path="/var/lib/kubelet/pods/3f463a32-7bc6-49a3-b65f-5358656350a0/volumes"
Jul 17 22:06:20.897674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.897595    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q"
Jul 17 22:06:20.898127 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.897588    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-e12150-95t2q" event={"ID":"771d4a38-47e9-4b4d-b59b-02dafd38e12a","Type":"ContainerDied","Data":"50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c"}
Jul 17 22:06:20.898127 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.897753    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50afe58e99c4af85c9f62827e1fe20906611759421e20e0c683e9e9d4b13638c"
Jul 17 22:06:20.899093 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.899077    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"
Jul 17 22:06:20.899177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.899093    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp" event={"ID":"673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f","Type":"ContainerDied","Data":"f9db84eee0fa1721965bf0bc12d3a5b92511fb5ae88457b74d952ef8042b579b"}
Jul 17 22:06:20.899177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.899158    2546 scope.go:117] "RemoveContainer" containerID="b22f831b7bc7ada4e8d2289480aea4ef697c04d6b63627cf83de3554f4c46f97"
Jul 17 22:06:20.914782 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.914763    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:20.919840 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:20.919821    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-7fbxp"]
Jul 17 22:06:21.906209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.906180    2546 generic.go:356] "Generic (PLEG): container finished" podID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerID="85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c" exitCode=0
Jul 17 22:06:21.906693 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.906258    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerDied","Data":"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"}
Jul 17 22:06:21.906693 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.906646    2546 scope.go:117] "RemoveContainer" containerID="85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"
Jul 17 22:06:21.907704 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.907677    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:21.907824 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.907770    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:21.907824 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:21.907803    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:06:22.536867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:22.536837    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" path="/var/lib/kubelet/pods/673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f/volumes"
Jul 17 22:06:22.914647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:22.914565    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerStarted","Data":"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"}
Jul 17 22:06:23.140412 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:23.140375    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:23.140614 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:23.140594    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Jul 17 22:06:23.140671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:23.140645    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Jul 17 22:06:24.140231 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:24.140195    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Jul 17 22:06:24.140714 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:24.140247    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Jul 17 22:06:25.139969 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:25.139933    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Jul 17 22:06:25.140150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:25.139984    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Jul 17 22:06:26.140247 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:26.140210    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body=
Jul 17 22:06:26.140647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:26.140263    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused"
Jul 17 22:06:27.143373 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:27.143347    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:27.143841 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:27.143814    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:27.144586 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:27.144569    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:06:31.530708 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:31.530682    2546 scope.go:117] "RemoveContainer" containerID="c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65"
Jul 17 22:06:31.962475 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:31.962448    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerStarted","Data":"b04df456b3983829da242f6a60f78fc4d03644f6c0c8d894dd9882406c7930a3"}
Jul 17 22:06:32.969771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:32.969737    2546 generic.go:356] "Generic (PLEG): container finished" podID="821e299b-62da-4421-8d93-ab51a6eaf603" containerID="b04df456b3983829da242f6a60f78fc4d03644f6c0c8d894dd9882406c7930a3" exitCode=6
Jul 17 22:06:32.969771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:32.969747    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerDied","Data":"b04df456b3983829da242f6a60f78fc4d03644f6c0c8d894dd9882406c7930a3"}
Jul 17 22:06:32.970232 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:32.969812    2546 scope.go:117] "RemoveContainer" containerID="c41e0c700d3d9fc8017b7dd242e76945c38a56f36070cb24e154e4f19c20af65"
Jul 17 22:06:32.970232 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:32.970071    2546 scope.go:117] "RemoveContainer" containerID="b04df456b3983829da242f6a60f78fc4d03644f6c0c8d894dd9882406c7930a3"
Jul 17 22:06:32.970311 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:32.970273    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7_odh-ai-gateway-infra(821e299b-62da-4421-8d93-ab51a6eaf603)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" podUID="821e299b-62da-4421-8d93-ab51a6eaf603"
Jul 17 22:06:32.996805 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:32.996781    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:34.108990 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.108966    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:34.229130 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.229070    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca\") pod \"821e299b-62da-4421-8d93-ab51a6eaf603\" (UID: \"821e299b-62da-4421-8d93-ab51a6eaf603\") "
Jul 17 22:06:34.229385 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.229363    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "821e299b-62da-4421-8d93-ab51a6eaf603" (UID: "821e299b-62da-4421-8d93-ab51a6eaf603"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:34.330026 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.329985    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/821e299b-62da-4421-8d93-ab51a6eaf603-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:34.983609 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.983530    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"
Jul 17 22:06:34.983783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.983530    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7" event={"ID":"821e299b-62da-4421-8d93-ab51a6eaf603","Type":"ContainerDied","Data":"45e57643dac5474a4af1a8b5d757f69e9b0b273fa355f8871045935c3c9c3049"}
Jul 17 22:06:34.983783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:34.983660    2546 scope.go:117] "RemoveContainer" containerID="b04df456b3983829da242f6a60f78fc4d03644f6c0c8d894dd9882406c7930a3"
Jul 17 22:06:35.003054 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:35.003031    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:35.007431 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:35.007411    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-7g4w7"]
Jul 17 22:06:36.536961 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:36.536926    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" path="/var/lib/kubelet/pods/821e299b-62da-4421-8d93-ab51a6eaf603/volumes"
Jul 17 22:06:38.213219 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213187    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213493    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213504    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213513    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213518    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213526    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213531    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213537    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213542    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213552    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213557    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213566    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213571    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213578    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213583    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213593    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213598    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213606    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="771d4a38-47e9-4b4d-b59b-02dafd38e12a" containerName="revoke-keys"
Jul 17 22:06:38.213601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213612    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="771d4a38-47e9-4b4d-b59b-02dafd38e12a" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213668    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213675    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213681    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213686    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213692    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213697    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="673e2e5b-6e9f-432f-8dcc-d5c3e5e8ad7f" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213703    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="771d4a38-47e9-4b4d-b59b-02dafd38e12a" containerName="revoke-keys"
Jul 17 22:06:38.214222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.213708    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:38.215673 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.215657    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:38.217959 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.217938    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:06:38.224848 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.224795    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:06:38.257701 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.257676    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg\" (UID: \"59aa4aa0-84fd-43ea-bc65-b84671a59f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:38.358336 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.358314    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg\" (UID: \"59aa4aa0-84fd-43ea-bc65-b84671a59f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:38.358927 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.358910    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg\" (UID: \"59aa4aa0-84fd-43ea-bc65-b84671a59f02\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:38.525799 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.525775    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:38.888290 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:38.888104    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:06:38.890858 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:38.890830    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59aa4aa0_84fd_43ea_bc65_b84671a59f02.slice/crio-f21f4ceeb0face1c316208ec94696cc6e06318178dfe6a26fd6156e0aa1b5337 WatchSource:0}: Error finding container f21f4ceeb0face1c316208ec94696cc6e06318178dfe6a26fd6156e0aa1b5337: Status 404 returned error can't find the container with id f21f4ceeb0face1c316208ec94696cc6e06318178dfe6a26fd6156e0aa1b5337
Jul 17 22:06:39.006063 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:39.006035    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerStarted","Data":"f21f4ceeb0face1c316208ec94696cc6e06318178dfe6a26fd6156e0aa1b5337"}
Jul 17 22:06:40.012667 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:40.012632    2546 generic.go:356] "Generic (PLEG): container finished" podID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerID="db7add2bc521447b39976cc85fc2c4c32b39a049942a28abcb3e66343be94955" exitCode=6
Jul 17 22:06:40.013208 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:40.012739    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerDied","Data":"db7add2bc521447b39976cc85fc2c4c32b39a049942a28abcb3e66343be94955"}
Jul 17 22:06:40.013208 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:40.012891    2546 scope.go:117] "RemoveContainer" containerID="db7add2bc521447b39976cc85fc2c4c32b39a049942a28abcb3e66343be94955"
Jul 17 22:06:41.019302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:41.019270    2546 generic.go:356] "Generic (PLEG): container finished" podID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerID="887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7" exitCode=6
Jul 17 22:06:41.019754 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:41.019343    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerDied","Data":"887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7"}
Jul 17 22:06:41.019754 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:41.019411    2546 scope.go:117] "RemoveContainer" containerID="db7add2bc521447b39976cc85fc2c4c32b39a049942a28abcb3e66343be94955"
Jul 17 22:06:41.019754 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:41.019649    2546 scope.go:117] "RemoveContainer" containerID="887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7"
Jul 17 22:06:41.019895 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:41.019865    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg_odh-ai-gateway-infra(59aa4aa0-84fd-43ea-bc65-b84671a59f02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02"
Jul 17 22:06:42.026533 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:42.026507    2546 scope.go:117] "RemoveContainer" containerID="887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7"
Jul 17 22:06:42.026930 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:42.026681    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg_odh-ai-gateway-infra(59aa4aa0-84fd-43ea-bc65-b84671a59f02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02"
Jul 17 22:06:42.816973 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:42.816937    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:06:42.817211 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:42.817184    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" containerID="cri-o://66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5" gracePeriod=30
Jul 17 22:06:43.164504 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.164424    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"]
Jul 17 22:06:43.164909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.164744    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:43.164909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.164755    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:43.164909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.164808    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f463a32-7bc6-49a3-b65f-5358656350a0" containerName="revoke-keys"
Jul 17 22:06:43.164909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.164817    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="821e299b-62da-4421-8d93-ab51a6eaf603" containerName="revoke-keys"
Jul 17 22:06:43.167236 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.167219    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:43.177801 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.177775    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"]
Jul 17 22:06:43.297585 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.297552    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92c219-wld6x\" (UID: \"0feed117-bc75-4c3e-9317-bd86a7a79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:43.398077 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.398055    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92c219-wld6x\" (UID: \"0feed117-bc75-4c3e-9317-bd86a7a79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:43.398604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.398587    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-92c219-wld6x\" (UID: \"0feed117-bc75-4c3e-9317-bd86a7a79983\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:43.478633 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.478572    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:43.605766 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:43.605742    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"]
Jul 17 22:06:43.606842 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:43.606819    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0feed117_bc75_4c3e_9317_bd86a7a79983.slice/crio-18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68 WatchSource:0}: Error finding container 18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68: Status 404 returned error can't find the container with id 18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68
Jul 17 22:06:44.039272 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:44.039198    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x" event={"ID":"0feed117-bc75-4c3e-9317-bd86a7a79983","Type":"ContainerStarted","Data":"a2019afbf78c088429fc8740807c44c50625e106e02815f28725b66b3213f457"}
Jul 17 22:06:44.039272 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:44.039257    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x" event={"ID":"0feed117-bc75-4c3e-9317-bd86a7a79983","Type":"ContainerStarted","Data":"18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68"}
Jul 17 22:06:44.054763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:44.054703    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x" podStartSLOduration=1.054689613 podStartE2EDuration="1.054689613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:44.052395505 +0000 UTC m=+2980.032765982" watchObservedRunningTime="2026-07-17 22:06:44.054689613 +0000 UTC m=+2980.035060089"
Jul 17 22:06:45.045840 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:45.045805    2546 generic.go:356] "Generic (PLEG): container finished" podID="0feed117-bc75-4c3e-9317-bd86a7a79983" containerID="a2019afbf78c088429fc8740807c44c50625e106e02815f28725b66b3213f457" exitCode=0
Jul 17 22:06:45.046201 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:45.045851    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x" event={"ID":"0feed117-bc75-4c3e-9317-bd86a7a79983","Type":"ContainerDied","Data":"a2019afbf78c088429fc8740807c44c50625e106e02815f28725b66b3213f457"}
Jul 17 22:06:45.686152 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:45.686109    2546 patch_prober.go:28] interesting pod/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Jul 17 22:06:45.686304 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:45.686193    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:06:46.177423 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:46.177401    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:46.221810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:46.221785    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca\") pod \"0feed117-bc75-4c3e-9317-bd86a7a79983\" (UID: \"0feed117-bc75-4c3e-9317-bd86a7a79983\") "
Jul 17 22:06:46.222115 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:46.222093    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0feed117-bc75-4c3e-9317-bd86a7a79983" (UID: "0feed117-bc75-4c3e-9317-bd86a7a79983"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:46.323345 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:46.323265    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0feed117-bc75-4c3e-9317-bd86a7a79983-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:47.058297 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:47.058267    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x"
Jul 17 22:06:47.058478 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:47.058311    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-92c219-wld6x" event={"ID":"0feed117-bc75-4c3e-9317-bd86a7a79983","Type":"ContainerDied","Data":"18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68"}
Jul 17 22:06:47.058478 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:47.058365    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="18b78e0652dd589aa18cdf771d7f9d3a29ec92e29289c92ff3d7648ef4d0da68"
Jul 17 22:06:48.064956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.064937    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:48.065334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.065313    2546 generic.go:356] "Generic (PLEG): container finished" podID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerID="66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5" exitCode=0
Jul 17 22:06:48.065399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.065382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerDied","Data":"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"}
Jul 17 22:06:48.065449 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.065437    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr" event={"ID":"c987f5f9-9079-4794-aa22-3f13c0f4e2fa","Type":"ContainerDied","Data":"6d52f40ba05b52099809d0f36b26722d3f162d02d7036f2f8fc7ed918b1db474"}
Jul 17 22:06:48.065500 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.065465    2546 scope.go:117] "RemoveContainer" containerID="66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"
Jul 17 22:06:48.073912 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.073895    2546 scope.go:117] "RemoveContainer" containerID="da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"
Jul 17 22:06:48.081802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.081778    2546 scope.go:117] "RemoveContainer" containerID="66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"
Jul 17 22:06:48.082092 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:48.082046    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5\": container with ID starting with 66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5 not found: ID does not exist" containerID="66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"
Jul 17 22:06:48.082092 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.082075    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5"} err="failed to get container status \"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5\": rpc error: code = NotFound desc = could not find container \"66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5\": container with ID starting with 66f0b84a067c3be0648a46131a7f348e57574ae468289a5ae3a7728ed356e4f5 not found: ID does not exist"
Jul 17 22:06:48.082242 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.082098    2546 scope.go:117] "RemoveContainer" containerID="da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"
Jul 17 22:06:48.082355 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:48.082337    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135\": container with ID starting with da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135 not found: ID does not exist" containerID="da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"
Jul 17 22:06:48.082402 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.082361    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135"} err="failed to get container status \"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135\": rpc error: code = NotFound desc = could not find container \"da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135\": container with ID starting with da5c2513471341393d4a2ce1f24a26f68b4703e8a65367ce1558fd2a1cd03135 not found: ID does not exist"
Jul 17 22:06:48.135340 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135315    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135359    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135386    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sthqn\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135445 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135411    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135540    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135575    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:48.135684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135592    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135682    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135911 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135698    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:48.135911 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket\") pod \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\" (UID: \"c987f5f9-9079-4794-aa22-3f13c0f4e2fa\") "
Jul 17 22:06:48.135911 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.135776    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:48.136068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.136050    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.136125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.136075    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.136125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.136091    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.136236 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.136128    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:48.136685 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.136662    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data" (OuterVolumeSpecName: "istio-data") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:48.137591 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.137568    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn" (OuterVolumeSpecName: "kube-api-access-sthqn") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "kube-api-access-sthqn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:48.137954 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.137929    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token" (OuterVolumeSpecName: "istio-token") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:06:48.138103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.138087    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:06:48.138103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.138092    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c987f5f9-9079-4794-aa22-3f13c0f4e2fa" (UID: "c987f5f9-9079-4794-aa22-3f13c0f4e2fa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:06:48.236374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236355    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.236477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236378    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.236477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236391    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.236477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236404    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.236477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236416    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:48.236477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:48.236429    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sthqn\" (UniqueName: \"kubernetes.io/projected/c987f5f9-9079-4794-aa22-3f13c0f4e2fa-kube-api-access-sthqn\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:49.071767 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.071741    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"
Jul 17 22:06:49.089893 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.089863    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:06:49.107912 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.107886    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-e12150-openshift-default-684fc6fd8c-9fcgr"]
Jul 17 22:06:49.143808 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.143783    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:06:49.144063 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144052    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.144114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144065    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.144114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144073    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0feed117-bc75-4c3e-9317-bd86a7a79983" containerName="revoke-keys"
Jul 17 22:06:49.144114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144079    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0feed117-bc75-4c3e-9317-bd86a7a79983" containerName="revoke-keys"
Jul 17 22:06:49.144114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144091    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.144114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144097    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.144270 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144173    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.144270 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144180    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0feed117-bc75-4c3e-9317-bd86a7a79983" containerName="revoke-keys"
Jul 17 22:06:49.144270 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.144185    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" containerName="istio-proxy"
Jul 17 22:06:49.146931 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.146916    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:06:49.153886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.153853    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:06:49.244823 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.244785    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5\" (UID: \"f27f3d5c-2687-4e70-80a4-a21d8e338391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:06:49.345850 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.345776    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5\" (UID: \"f27f3d5c-2687-4e70-80a4-a21d8e338391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:06:49.346329 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.346312    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5\" (UID: \"f27f3d5c-2687-4e70-80a4-a21d8e338391\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:06:49.458366 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.458341    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:06:49.590322 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:49.590293    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:06:49.591711 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:06:49.591684    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf27f3d5c_2687_4e70_80a4_a21d8e338391.slice/crio-50d962ec7beb8a168a97fc8d0b3be65391a829618341b7ccdbb652e1e8252c38 WatchSource:0}: Error finding container 50d962ec7beb8a168a97fc8d0b3be65391a829618341b7ccdbb652e1e8252c38: Status 404 returned error can't find the container with id 50d962ec7beb8a168a97fc8d0b3be65391a829618341b7ccdbb652e1e8252c38
Jul 17 22:06:50.078279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:50.078244    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerStarted","Data":"3a2303595793625a0397a2e76fa20e484fc75955a5bea58f8f138656986444c8"}
Jul 17 22:06:50.078648 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:50.078304    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerStarted","Data":"50d962ec7beb8a168a97fc8d0b3be65391a829618341b7ccdbb652e1e8252c38"}
Jul 17 22:06:50.094008 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:50.093967    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" podStartSLOduration=1.093951508 podStartE2EDuration="1.093951508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:50.092441919 +0000 UTC m=+2986.072812396" watchObservedRunningTime="2026-07-17 22:06:50.093951508 +0000 UTC m=+2986.074321984"
Jul 17 22:06:50.536048 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:50.536015    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c987f5f9-9079-4794-aa22-3f13c0f4e2fa" path="/var/lib/kubelet/pods/c987f5f9-9079-4794-aa22-3f13c0f4e2fa/volumes"
Jul 17 22:06:51.085375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:51.085337    2546 generic.go:356] "Generic (PLEG): container finished" podID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerID="3a2303595793625a0397a2e76fa20e484fc75955a5bea58f8f138656986444c8" exitCode=6
Jul 17 22:06:51.085773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:51.085426    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerDied","Data":"3a2303595793625a0397a2e76fa20e484fc75955a5bea58f8f138656986444c8"}
Jul 17 22:06:51.085773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:51.085585    2546 scope.go:117] "RemoveContainer" containerID="3a2303595793625a0397a2e76fa20e484fc75955a5bea58f8f138656986444c8"
Jul 17 22:06:52.092378 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:52.092342    2546 generic.go:356] "Generic (PLEG): container finished" podID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerID="ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9" exitCode=6
Jul 17 22:06:52.092865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:52.092389    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerDied","Data":"ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9"}
Jul 17 22:06:52.092865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:52.092444    2546 scope.go:117] "RemoveContainer" containerID="3a2303595793625a0397a2e76fa20e484fc75955a5bea58f8f138656986444c8"
Jul 17 22:06:52.092865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:52.092616    2546 scope.go:117] "RemoveContainer" containerID="ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9"
Jul 17 22:06:52.092865 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:52.092844    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5_odh-ai-gateway-infra(f27f3d5c-2687-4e70-80a4-a21d8e338391)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391"
Jul 17 22:06:53.099810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:53.099785    2546 scope.go:117] "RemoveContainer" containerID="ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9"
Jul 17 22:06:53.100164 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:53.099960    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5_odh-ai-gateway-infra(f27f3d5c-2687-4e70-80a4-a21d8e338391)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391"
Jul 17 22:06:55.531432 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:55.531401    2546 scope.go:117] "RemoveContainer" containerID="887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7"
Jul 17 22:06:56.116430 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:56.116340    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerStarted","Data":"104962dbfff5cc03eea3d042bf4ec14e648b2ca7bb3a5279c4e82e1b6767c922"}
Jul 17 22:06:56.134515 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:56.134471    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" podStartSLOduration=18.134457044 podStartE2EDuration="18.134457044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:06:56.131630263 +0000 UTC m=+2992.112000740" watchObservedRunningTime="2026-07-17 22:06:56.134457044 +0000 UTC m=+2992.114827520"
Jul 17 22:06:57.124570 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:57.124535    2546 generic.go:356] "Generic (PLEG): container finished" podID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerID="104962dbfff5cc03eea3d042bf4ec14e648b2ca7bb3a5279c4e82e1b6767c922" exitCode=6
Jul 17 22:06:57.125081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:57.124616    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerDied","Data":"104962dbfff5cc03eea3d042bf4ec14e648b2ca7bb3a5279c4e82e1b6767c922"}
Jul 17 22:06:57.125081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:57.124688    2546 scope.go:117] "RemoveContainer" containerID="887b6066869537779d43f8372b570a147b32752e22ec1990d79fe67e70fb88b7"
Jul 17 22:06:57.125081 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:57.125014    2546 scope.go:117] "RemoveContainer" containerID="104962dbfff5cc03eea3d042bf4ec14e648b2ca7bb3a5279c4e82e1b6767c922"
Jul 17 22:06:57.125264 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:06:57.125245    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg_odh-ai-gateway-infra(59aa4aa0-84fd-43ea-bc65-b84671a59f02)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02"
Jul 17 22:06:57.149080 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:57.149056    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:06:58.266184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:58.266155    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:58.418259 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:58.418185    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca\") pod \"59aa4aa0-84fd-43ea-bc65-b84671a59f02\" (UID: \"59aa4aa0-84fd-43ea-bc65-b84671a59f02\") "
Jul 17 22:06:58.418509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:58.418488    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "59aa4aa0-84fd-43ea-bc65-b84671a59f02" (UID: "59aa4aa0-84fd-43ea-bc65-b84671a59f02"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:06:58.519028 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:58.518999    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59aa4aa0-84fd-43ea-bc65-b84671a59f02-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:06:59.139308 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:59.139274    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg" event={"ID":"59aa4aa0-84fd-43ea-bc65-b84671a59f02","Type":"ContainerDied","Data":"f21f4ceeb0face1c316208ec94696cc6e06318178dfe6a26fd6156e0aa1b5337"}
Jul 17 22:06:59.139473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:59.139335    2546 scope.go:117] "RemoveContainer" containerID="104962dbfff5cc03eea3d042bf4ec14e648b2ca7bb3a5279c4e82e1b6767c922"
Jul 17 22:06:59.139473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:59.139285    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"
Jul 17 22:06:59.156951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:59.156926    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:06:59.163663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:06:59.163645    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-kg7tg"]
Jul 17 22:07:00.535783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.535749    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" path="/var/lib/kubelet/pods/59aa4aa0-84fd-43ea-bc65-b84671a59f02/volumes"
Jul 17 22:07:00.964008 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.963934    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:00.964230 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964218    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964232    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964252    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964257    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964368 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964307    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964368 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964314    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.964368 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.964320    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:00.967457 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.967441    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:00.974199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:00.974172    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:01.036374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:01.036352    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk\" (UID: \"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:01.136910 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:01.136881    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk\" (UID: \"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:01.137399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:01.137381    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk\" (UID: \"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:01.278503 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:01.278481    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:01.405942 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:01.405919    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:01.407356 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:01.407330    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ddf972b_8ec5_465f_96c6_4b94eb7bdbbf.slice/crio-019afd9a7819ff3bc0a920fc35ced4baecf491aca57b5a5bb34de780f0192120 WatchSource:0}: Error finding container 019afd9a7819ff3bc0a920fc35ced4baecf491aca57b5a5bb34de780f0192120: Status 404 returned error can't find the container with id 019afd9a7819ff3bc0a920fc35ced4baecf491aca57b5a5bb34de780f0192120
Jul 17 22:07:02.157300 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:02.157268    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerID="9e92e0c64248fc08da786eccc83b193beec83b99ca841e949de11f544e1373a4" exitCode=6
Jul 17 22:07:02.157759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:02.157355    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerDied","Data":"9e92e0c64248fc08da786eccc83b193beec83b99ca841e949de11f544e1373a4"}
Jul 17 22:07:02.157759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:02.157402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerStarted","Data":"019afd9a7819ff3bc0a920fc35ced4baecf491aca57b5a5bb34de780f0192120"}
Jul 17 22:07:02.157759 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:02.157471    2546 scope.go:117] "RemoveContainer" containerID="9e92e0c64248fc08da786eccc83b193beec83b99ca841e949de11f544e1373a4"
Jul 17 22:07:03.166944 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:03.166855    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerID="95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327" exitCode=6
Jul 17 22:07:03.167420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:03.166934    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerDied","Data":"95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327"}
Jul 17 22:07:03.167420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:03.166998    2546 scope.go:117] "RemoveContainer" containerID="9e92e0c64248fc08da786eccc83b193beec83b99ca841e949de11f544e1373a4"
Jul 17 22:07:03.167420 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:03.167177    2546 scope.go:117] "RemoveContainer" containerID="95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327"
Jul 17 22:07:03.167420 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:03.167401    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk_odh-ai-gateway-infra(7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf"
Jul 17 22:07:04.174752 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:04.174696    2546 scope.go:117] "RemoveContainer" containerID="95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327"
Jul 17 22:07:04.175180 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:04.174882    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk_odh-ai-gateway-infra(7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf"
Jul 17 22:07:04.536480 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:04.536451    2546 scope.go:117] "RemoveContainer" containerID="ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9"
Jul 17 22:07:04.721971 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:04.721941    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:07:04.731282 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:04.731260    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:07:05.180405 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.180378    2546 generic.go:356] "Generic (PLEG): container finished" podID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerID="b3794666b14b3787c269f5439174b3d6a4547ea7405c8648128ffdb3eb057477" exitCode=6
Jul 17 22:07:05.180768 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.180457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerDied","Data":"b3794666b14b3787c269f5439174b3d6a4547ea7405c8648128ffdb3eb057477"}
Jul 17 22:07:05.180768 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.180533    2546 scope.go:117] "RemoveContainer" containerID="ead5be6d22ba84d506c3ce5c43797fe1f49780aa963a0875598df26c281a34c9"
Jul 17 22:07:05.180874 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.180857    2546 scope.go:117] "RemoveContainer" containerID="b3794666b14b3787c269f5439174b3d6a4547ea7405c8648128ffdb3eb057477"
Jul 17 22:07:05.181042 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:05.181020    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5_odh-ai-gateway-infra(f27f3d5c-2687-4e70-80a4-a21d8e338391)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391"
Jul 17 22:07:05.738664 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.738631    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:05.739005 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.738990    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:05.739005 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.739006    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="59aa4aa0-84fd-43ea-bc65-b84671a59f02" containerName="revoke-keys"
Jul 17 22:07:05.743313 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.743293    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:05.749578 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.749550    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:05.876821 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.876788    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb\" (UID: \"f246026c-b172-4ab5-b3c5-774b27eaa26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:05.977760 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.977710    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb\" (UID: \"f246026c-b172-4ab5-b3c5-774b27eaa26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:05.978300 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:05.978281    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb\" (UID: \"f246026c-b172-4ab5-b3c5-774b27eaa26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:06.056794 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.056738    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:06.183946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.183923    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:06.185944 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:06.185921    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf246026c_b172_4ab5_b3c5_774b27eaa26a.slice/crio-1767a50508e8f590787c9ba097d57e13cedafad4a6ef938b11829773d2dbba67 WatchSource:0}: Error finding container 1767a50508e8f590787c9ba097d57e13cedafad4a6ef938b11829773d2dbba67: Status 404 returned error can't find the container with id 1767a50508e8f590787c9ba097d57e13cedafad4a6ef938b11829773d2dbba67
Jul 17 22:07:06.207886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.207862    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:07:06.339948 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.339926    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:07:06.481415 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.481393    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca\") pod \"f27f3d5c-2687-4e70-80a4-a21d8e338391\" (UID: \"f27f3d5c-2687-4e70-80a4-a21d8e338391\") "
Jul 17 22:07:06.481696 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.481676    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f27f3d5c-2687-4e70-80a4-a21d8e338391" (UID: "f27f3d5c-2687-4e70-80a4-a21d8e338391"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:06.582383 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:06.582332    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f27f3d5c-2687-4e70-80a4-a21d8e338391-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:07.195508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.195472    2546 generic.go:356] "Generic (PLEG): container finished" podID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerID="b0e9f2924fe3bd328144d9a38fe17e7b550ab0df5faea017e4be6581a7cfd6f9" exitCode=6
Jul 17 22:07:07.195971 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.195564    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerDied","Data":"b0e9f2924fe3bd328144d9a38fe17e7b550ab0df5faea017e4be6581a7cfd6f9"}
Jul 17 22:07:07.195971 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.195628    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerStarted","Data":"1767a50508e8f590787c9ba097d57e13cedafad4a6ef938b11829773d2dbba67"}
Jul 17 22:07:07.195971 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.195768    2546 scope.go:117] "RemoveContainer" containerID="b0e9f2924fe3bd328144d9a38fe17e7b550ab0df5faea017e4be6581a7cfd6f9"
Jul 17 22:07:07.197061 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.197040    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"
Jul 17 22:07:07.197061 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.197046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5" event={"ID":"f27f3d5c-2687-4e70-80a4-a21d8e338391","Type":"ContainerDied","Data":"50d962ec7beb8a168a97fc8d0b3be65391a829618341b7ccdbb652e1e8252c38"}
Jul 17 22:07:07.197215 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.197099    2546 scope.go:117] "RemoveContainer" containerID="b3794666b14b3787c269f5439174b3d6a4547ea7405c8648128ffdb3eb057477"
Jul 17 22:07:07.227077 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.227051    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:07:07.228973 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:07.228954    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-j6nm5"]
Jul 17 22:07:08.206194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.206101    2546 generic.go:356] "Generic (PLEG): container finished" podID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerID="063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c" exitCode=6
Jul 17 22:07:08.206194 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.206161    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerDied","Data":"063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c"}
Jul 17 22:07:08.206707 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.206235    2546 scope.go:117] "RemoveContainer" containerID="b0e9f2924fe3bd328144d9a38fe17e7b550ab0df5faea017e4be6581a7cfd6f9"
Jul 17 22:07:08.206707 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.206437    2546 scope.go:117] "RemoveContainer" containerID="063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c"
Jul 17 22:07:08.206707 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:08.206617    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb_odh-ai-gateway-infra(f246026c-b172-4ab5-b3c5-774b27eaa26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a"
Jul 17 22:07:08.536325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.536298    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" path="/var/lib/kubelet/pods/f27f3d5c-2687-4e70-80a4-a21d8e338391/volumes"
Jul 17 22:07:08.904852 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.904779    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:07:08.905009 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:08.904983    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" containerID="cri-o://944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a" gracePeriod=30
Jul 17 22:07:09.213488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.213414    2546 scope.go:117] "RemoveContainer" containerID="063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c"
Jul 17 22:07:09.213868 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:09.213610    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb_odh-ai-gateway-infra(f246026c-b172-4ab5-b3c5-774b27eaa26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a"
Jul 17 22:07:09.730233 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730206    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:07:09.730522 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730508    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730522 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730523    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730537    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730543    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730552    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730560    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730796 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730644    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.730796 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.730654    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:09.735069 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.735052    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.737698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.737678    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-2a645a-openshift-default-dockercfg-6w464\""
Jul 17 22:07:09.745222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.745198    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:07:09.909389 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909360    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909402    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909418    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909495    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909543 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909523    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909678 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909548    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909678 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909580    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-95zd9\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909678 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909603    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:09.909678 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:09.909619    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.010755 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010709    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.010895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010781    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.010895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010812    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.010895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010860    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010899    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010928    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010960    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-95zd9\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.010996    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011023    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011184    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011254    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011490 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011453    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011490 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011467    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.011625 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.011500    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.013301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.013280    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.013556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.013537    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.020949 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.020925    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.021421 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.021398    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-95zd9\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9\") pod \"e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") " pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.045866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.045841    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:10.188525 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.188498    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:07:10.189713 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:10.189676    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b663148_c97e_43be_a3d1_fb6183f66014.slice/crio-833b24d2d01d3909e32ca4c509469b431f770a2fdf2a17d6beff7dab9855f00e WatchSource:0}: Error finding container 833b24d2d01d3909e32ca4c509469b431f770a2fdf2a17d6beff7dab9855f00e: Status 404 returned error can't find the container with id 833b24d2d01d3909e32ca4c509469b431f770a2fdf2a17d6beff7dab9855f00e
Jul 17 22:07:10.192028 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.192000    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:10.192096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.192058    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:10.192096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.192086    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:10.219148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:10.219125    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerStarted","Data":"833b24d2d01d3909e32ca4c509469b431f770a2fdf2a17d6beff7dab9855f00e"}
Jul 17 22:07:11.225650 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:11.225612    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerStarted","Data":"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"}
Jul 17 22:07:11.249149 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:11.249101    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podStartSLOduration=2.249086578 podStartE2EDuration="2.249086578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:11.248194243 +0000 UTC m=+3007.228564777" watchObservedRunningTime="2026-07-17 22:07:11.249086578 +0000 UTC m=+3007.229457055"
Jul 17 22:07:12.046821 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.046782    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:12.051574 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.051551    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:12.145250 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.145219    2546 patch_prober.go:28] interesting pod/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 17 22:07:12.145377 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.145288    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 17 22:07:12.231324 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.231294    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:12.232296 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:12.232279    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:14.150012 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.149988    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:07:14.244877 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.244853    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.244889    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.244919    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.244936    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.244972    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245014    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245063    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rf7sf\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245093    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245140    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy\") pod \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\" (UID: \"3e8e7949-5070-4f34-bbdb-b4df824fbd23\") "
Jul 17 22:07:14.245338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245171    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:14.245338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245190    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:14.245592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245353    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:14.245592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245456    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.245592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245474    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.245592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245489    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.245592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245527    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:14.245977 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.245956    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data" (OuterVolumeSpecName: "istio-data") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:14.246495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.246474    2546 generic.go:356] "Generic (PLEG): container finished" podID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerID="944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a" exitCode=0
Jul 17 22:07:14.246756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.246591    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"
Jul 17 22:07:14.246756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.246623    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerDied","Data":"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"}
Jul 17 22:07:14.246756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.246667    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp" event={"ID":"3e8e7949-5070-4f34-bbdb-b4df824fbd23","Type":"ContainerDied","Data":"38e546c89585589ac4d5d48481661bb3a236c75d32a290402f17ca8d3545a10e"}
Jul 17 22:07:14.246756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.246755    2546 scope.go:117] "RemoveContainer" containerID="944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"
Jul 17 22:07:14.248063 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.248043    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:07:14.248169 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.248041    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token" (OuterVolumeSpecName: "istio-token") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:07:14.248347 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.248322    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf" (OuterVolumeSpecName: "kube-api-access-rf7sf") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "kube-api-access-rf7sf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:07:14.248572 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.248540    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "3e8e7949-5070-4f34-bbdb-b4df824fbd23" (UID: "3e8e7949-5070-4f34-bbdb-b4df824fbd23"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:07:14.264069 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.264047    2546 scope.go:117] "RemoveContainer" containerID="85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"
Jul 17 22:07:14.271853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.271827    2546 scope.go:117] "RemoveContainer" containerID="944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"
Jul 17 22:07:14.272105 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:14.272086    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a\": container with ID starting with 944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a not found: ID does not exist" containerID="944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"
Jul 17 22:07:14.272162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.272114    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a"} err="failed to get container status \"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a\": rpc error: code = NotFound desc = could not find container \"944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a\": container with ID starting with 944931764adf18ec33791d5ed85cb70aef210cc4632da10a38f876dd6487ad9a not found: ID does not exist"
Jul 17 22:07:14.272162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.272132    2546 scope.go:117] "RemoveContainer" containerID="85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"
Jul 17 22:07:14.272369 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:14.272350    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c\": container with ID starting with 85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c not found: ID does not exist" containerID="85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"
Jul 17 22:07:14.272425 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.272379    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c"} err="failed to get container status \"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c\": rpc error: code = NotFound desc = could not find container \"85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c\": container with ID starting with 85d44ab543b3b0e58a7e9df8d0e4db49dc8f690b7b9639903611a78a50115e6c not found: ID does not exist"
Jul 17 22:07:14.346225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346199    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.346225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346219    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.346225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346228    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.346439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346236    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.346439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346246    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf7sf\" (UniqueName: \"kubernetes.io/projected/3e8e7949-5070-4f34-bbdb-b4df824fbd23-kube-api-access-rf7sf\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.346439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.346254    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3e8e7949-5070-4f34-bbdb-b4df824fbd23-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:14.564739 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.564692    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:07:14.570988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:14.570964    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-92c219-openshift-default-b758d97d9-bh2fp"]
Jul 17 22:07:16.536767 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:16.536702    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" path="/var/lib/kubelet/pods/3e8e7949-5070-4f34-bbdb-b4df824fbd23/volumes"
Jul 17 22:07:18.242798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.242765    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:07:18.243211 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243197    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.243255 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243214    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.243255 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243244    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.243255 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243253    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.243344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243333    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.243381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243344    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f27f3d5c-2687-4e70-80a4-a21d8e338391" containerName="revoke-keys"
Jul 17 22:07:18.243381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.243353    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e8e7949-5070-4f34-bbdb-b4df824fbd23" containerName="istio-proxy"
Jul 17 22:07:18.247910 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.247892    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.254507 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.254487    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b96f25-openshift-default-dockercfg-bvjzc\""
Jul 17 22:07:18.270967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.270943    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:07:18.371828 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.371798    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.371981 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.371837    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xlk\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.371981 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.371863    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.371981 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.371955    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.372103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.372000    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.372103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.372050    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.372103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.372069    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.372103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.372095    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.372226 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.372113    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473322 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473289    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473331    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xlk\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473471    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473532    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473578    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473682    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473753    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473787    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473814    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473818    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.473868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.473836    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.474214 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.474094    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.474214 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.474157    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.474306 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.474242    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.475865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.475845    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.476313 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.476292    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.482589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.482550    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xlk\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.482805 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.482710    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token\") pod \"e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") " pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.531067 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.531009    2546 scope.go:117] "RemoveContainer" containerID="95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327"
Jul 17 22:07:18.557866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.557845    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:18.701526 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.701503    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:07:18.702430 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:18.702402    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod213f2544_8a42_406f_9f13_792c711d736e.slice/crio-0ee6aa55b90afd2f38012a6115794e6b68a341698b55cef015293d739106e629 WatchSource:0}: Error finding container 0ee6aa55b90afd2f38012a6115794e6b68a341698b55cef015293d739106e629: Status 404 returned error can't find the container with id 0ee6aa55b90afd2f38012a6115794e6b68a341698b55cef015293d739106e629
Jul 17 22:07:18.704508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.704479    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:18.704591 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.704537    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:18.704591 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:18.704564    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:19.279566 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.279524    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerStarted","Data":"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"}
Jul 17 22:07:19.280094 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.279601    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerStarted","Data":"0ee6aa55b90afd2f38012a6115794e6b68a341698b55cef015293d739106e629"}
Jul 17 22:07:19.282947 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.282907    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerStarted","Data":"000e745430972a9a7d0f8c2498a8379514ec2838c1e35a8f2e2971d5a25daf0b"}
Jul 17 22:07:19.307522 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.307461    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podStartSLOduration=1.307446421 podStartE2EDuration="1.307446421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:19.305495161 +0000 UTC m=+3015.285865639" watchObservedRunningTime="2026-07-17 22:07:19.307446421 +0000 UTC m=+3015.287816898"
Jul 17 22:07:19.325404 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.325365    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" podStartSLOduration=19.325353469 podStartE2EDuration="19.325353469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:19.323641881 +0000 UTC m=+3015.304012361" watchObservedRunningTime="2026-07-17 22:07:19.325353469 +0000 UTC m=+3015.305723986"
Jul 17 22:07:19.561103 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.560976    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:19.569375 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:19.569112    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:20.290105 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.290077    2546 generic.go:356] "Generic (PLEG): container finished" podID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerID="000e745430972a9a7d0f8c2498a8379514ec2838c1e35a8f2e2971d5a25daf0b" exitCode=6
Jul 17 22:07:20.290601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.290151    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerDied","Data":"000e745430972a9a7d0f8c2498a8379514ec2838c1e35a8f2e2971d5a25daf0b"}
Jul 17 22:07:20.290601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.290206    2546 scope.go:117] "RemoveContainer" containerID="95f007cd398182433df4cb92ef3b3e76a1f03fb847da23130990976bc3279327"
Jul 17 22:07:20.290601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.290404    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:20.290797 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.290665    2546 scope.go:117] "RemoveContainer" containerID="000e745430972a9a7d0f8c2498a8379514ec2838c1e35a8f2e2971d5a25daf0b"
Jul 17 22:07:20.290915 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:20.290875    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk_odh-ai-gateway-infra(7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf"
Jul 17 22:07:20.291641 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.291622    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:20.341429 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:20.341400    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:21.452527 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:21.452499    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:21.607846 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:21.607762    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca\") pod \"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf\" (UID: \"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf\") "
Jul 17 22:07:21.608065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:21.608045    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" (UID: "7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:21.708549 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:21.708518    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:22.304905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.304874    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"
Jul 17 22:07:22.305118 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.304917    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk" event={"ID":"7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf","Type":"ContainerDied","Data":"019afd9a7819ff3bc0a920fc35ced4baecf491aca57b5a5bb34de780f0192120"}
Jul 17 22:07:22.305118 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.304975    2546 scope.go:117] "RemoveContainer" containerID="000e745430972a9a7d0f8c2498a8379514ec2838c1e35a8f2e2971d5a25daf0b"
Jul 17 22:07:22.328570 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.328547    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:22.332900 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.332874    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jdhnk"]
Jul 17 22:07:22.531780 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.531749    2546 scope.go:117] "RemoveContainer" containerID="063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c"
Jul 17 22:07:22.536579 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:22.536556    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" path="/var/lib/kubelet/pods/7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf/volumes"
Jul 17 22:07:23.312992 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:23.312961    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerStarted","Data":"1a3e7a405de15533a6458cde4d378db4d30b3a3d77a9eeda0edf21ba9ef4c5cc"}
Jul 17 22:07:23.328285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:23.328235    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" podStartSLOduration=18.32822092 podStartE2EDuration="18.32822092s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:23.327796063 +0000 UTC m=+3019.308166540" watchObservedRunningTime="2026-07-17 22:07:23.32822092 +0000 UTC m=+3019.308591396"
Jul 17 22:07:24.322456 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:24.322423    2546 generic.go:356] "Generic (PLEG): container finished" podID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerID="1a3e7a405de15533a6458cde4d378db4d30b3a3d77a9eeda0edf21ba9ef4c5cc" exitCode=6
Jul 17 22:07:24.322950 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:24.322479    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerDied","Data":"1a3e7a405de15533a6458cde4d378db4d30b3a3d77a9eeda0edf21ba9ef4c5cc"}
Jul 17 22:07:24.322950 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:24.322530    2546 scope.go:117] "RemoveContainer" containerID="063d0ddd3491f8f50a1dffd6d0f7f0fcd27cfad27840c9949bf8818a352f203c"
Jul 17 22:07:24.322950 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:24.322834    2546 scope.go:117] "RemoveContainer" containerID="1a3e7a405de15533a6458cde4d378db4d30b3a3d77a9eeda0edf21ba9ef4c5cc"
Jul 17 22:07:24.323124 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:24.323064    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb_odh-ai-gateway-infra(f246026c-b172-4ab5-b3c5-774b27eaa26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a"
Jul 17 22:07:24.337790 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:24.337765    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:25.459985 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:25.459961    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:25.532331 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:25.532302    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca\") pod \"f246026c-b172-4ab5-b3c5-774b27eaa26a\" (UID: \"f246026c-b172-4ab5-b3c5-774b27eaa26a\") "
Jul 17 22:07:25.532649 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:25.532625    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f246026c-b172-4ab5-b3c5-774b27eaa26a" (UID: "f246026c-b172-4ab5-b3c5-774b27eaa26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:25.633038 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:25.632969    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f246026c-b172-4ab5-b3c5-774b27eaa26a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:26.336447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.336418    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b663148-c97e-43be-a3d1-fb6183f66014" containerID="f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620" exitCode=0
Jul 17 22:07:26.336657 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.336492    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerDied","Data":"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"}
Jul 17 22:07:26.336968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.336947    2546 scope.go:117] "RemoveContainer" containerID="f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"
Jul 17 22:07:26.337774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.337745    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb" event={"ID":"f246026c-b172-4ab5-b3c5-774b27eaa26a","Type":"ContainerDied","Data":"1767a50508e8f590787c9ba097d57e13cedafad4a6ef938b11829773d2dbba67"}
Jul 17 22:07:26.337862 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.337787    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"
Jul 17 22:07:26.337862 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.337791    2546 scope.go:117] "RemoveContainer" containerID="1a3e7a405de15533a6458cde4d378db4d30b3a3d77a9eeda0edf21ba9ef4c5cc"
Jul 17 22:07:26.337991 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.337962    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:26.338098 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.338050    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:26.338170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.338105    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:26.375583 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.375562    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:26.379945 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.379926    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-cv4kb"]
Jul 17 22:07:26.536167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:26.536135    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" path="/var/lib/kubelet/pods/f246026c-b172-4ab5-b3c5-774b27eaa26a/volumes"
Jul 17 22:07:27.232284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:27.232252    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:27.345014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:27.344979    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerStarted","Data":"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"}
Jul 17 22:07:28.046777 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:28.046746    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:28.047888 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:28.047857    2546 patch_prober.go:28] interesting pod/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Jul 17 22:07:28.048036 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:28.047927    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Jul 17 22:07:29.047191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:29.047158    2546 patch_prober.go:28] interesting pod/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Jul 17 22:07:29.047661 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:29.047212    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Jul 17 22:07:30.046675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:30.046643    2546 patch_prober.go:28] interesting pod/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Jul 17 22:07:30.046887 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:30.046693    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Jul 17 22:07:31.047099 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:31.047063    2546 patch_prober.go:28] interesting pod/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused" start-of-body=
Jul 17 22:07:31.047480 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:31.047116    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.142:15021/healthz/ready\": dial tcp 10.133.0.142:15021: connect: connection refused"
Jul 17 22:07:32.050351 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:32.050321    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:32.050849 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:32.050583    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:32.051259 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:32.051237    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:07:36.332907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:36.332865    2546 patch_prober.go:28] interesting pod/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 17 22:07:36.333377 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:36.332922    2546 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 17 22:07:37.408515 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.408482    2546 generic.go:356] "Generic (PLEG): container finished" podID="213f2544-8a42-406f-9f13-792c711d736e" containerID="41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad" exitCode=0
Jul 17 22:07:37.408861 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.408526    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerDied","Data":"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"}
Jul 17 22:07:37.408933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.408918    2546 scope.go:117] "RemoveContainer" containerID="41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"
Jul 17 22:07:37.409898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.409873    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:37.409956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.409941    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:37.409999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:37.409973    2546 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 17 22:07:38.345539 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345505    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:38.345832 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345819    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345833    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345845    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345850    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345856    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345862    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345870    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345876    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345891    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.345892 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345896    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345902    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345908    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345952    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345959    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345965    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:38.346171 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.345971    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:38.347908 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.347889    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:38.350302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.350265    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:07:38.356635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.356609    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:38.416091 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.416061    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerStarted","Data":"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"}
Jul 17 22:07:38.437554 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.437521    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85\" (UID: \"8ee9fc33-9add-4dda-85ef-20534fe40b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:38.538220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.538168    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85\" (UID: \"8ee9fc33-9add-4dda-85ef-20534fe40b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:38.538773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.538752    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85\" (UID: \"8ee9fc33-9add-4dda-85ef-20534fe40b45\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:38.558168 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.558145    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:38.558460 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.558435    2546 patch_prober.go:28] interesting pod/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Jul 17 22:07:38.558553 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.558489    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Jul 17 22:07:38.659895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.659799    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:38.830783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:38.830754    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:38.831952 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:38.831925    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee9fc33_9add_4dda_85ef_20534fe40b45.slice/crio-ebd158154ed28c9287144acbc188aa59b6e091aa5db04c5b3da7036b6e2c171c WatchSource:0}: Error finding container ebd158154ed28c9287144acbc188aa59b6e091aa5db04c5b3da7036b6e2c171c: Status 404 returned error can't find the container with id ebd158154ed28c9287144acbc188aa59b6e091aa5db04c5b3da7036b6e2c171c
Jul 17 22:07:39.188525 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.188501    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:07:39.188894 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.188881    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="f246026c-b172-4ab5-b3c5-774b27eaa26a" containerName="revoke-keys"
Jul 17 22:07:39.188952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.188897    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ddf972b-8ec5-465f-96c6-4b94eb7bdbbf" containerName="revoke-keys"
Jul 17 22:07:39.190666 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.190652    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:39.199947 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.199919    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:07:39.244770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.244740    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj\" (UID: \"72ea0f90-bd32-4029-8294-48cfd9b15fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:39.345437 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.345360    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj\" (UID: \"72ea0f90-bd32-4029-8294-48cfd9b15fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:39.345997 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.345977    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj\" (UID: \"72ea0f90-bd32-4029-8294-48cfd9b15fa6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:39.424790 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.424753    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerStarted","Data":"17203933a0c0f46cf364d52ad24c86a92e4da2a2314e0cfc1c1284b47f051679"}
Jul 17 22:07:39.425211 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.424810    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerStarted","Data":"ebd158154ed28c9287144acbc188aa59b6e091aa5db04c5b3da7036b6e2c171c"}
Jul 17 22:07:39.442748 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.442689    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" podStartSLOduration=1.442674792 podStartE2EDuration="1.442674792s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:39.440604715 +0000 UTC m=+3035.420975194" watchObservedRunningTime="2026-07-17 22:07:39.442674792 +0000 UTC m=+3035.423045270"
Jul 17 22:07:39.501934 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.501897    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:39.560284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.560256    2546 patch_prober.go:28] interesting pod/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Jul 17 22:07:39.560515 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.560488    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Jul 17 22:07:39.650220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:39.650194    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:07:39.651644 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:07:39.651606    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72ea0f90_bd32_4029_8294_48cfd9b15fa6.slice/crio-d424cd47115be9b23b4ca8ed7ff7a1ccaceb95ac39c6cb7e48efdc8027057791 WatchSource:0}: Error finding container d424cd47115be9b23b4ca8ed7ff7a1ccaceb95ac39c6cb7e48efdc8027057791: Status 404 returned error can't find the container with id d424cd47115be9b23b4ca8ed7ff7a1ccaceb95ac39c6cb7e48efdc8027057791
Jul 17 22:07:40.431853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.431818    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerStarted","Data":"66f7d2a0bd2c0b61ff482afef6c63373f7d2af67a074f962af91231225e3934d"}
Jul 17 22:07:40.432345 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.431874    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerStarted","Data":"d424cd47115be9b23b4ca8ed7ff7a1ccaceb95ac39c6cb7e48efdc8027057791"}
Jul 17 22:07:40.433423 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.433397    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerID="17203933a0c0f46cf364d52ad24c86a92e4da2a2314e0cfc1c1284b47f051679" exitCode=6
Jul 17 22:07:40.433548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.433440    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerDied","Data":"17203933a0c0f46cf364d52ad24c86a92e4da2a2314e0cfc1c1284b47f051679"}
Jul 17 22:07:40.433644 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.433628    2546 scope.go:117] "RemoveContainer" containerID="17203933a0c0f46cf364d52ad24c86a92e4da2a2314e0cfc1c1284b47f051679"
Jul 17 22:07:40.454191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.454148    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" podStartSLOduration=1.4541327370000001 podStartE2EDuration="1.454132737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:07:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:07:40.452398058 +0000 UTC m=+3036.432768538" watchObservedRunningTime="2026-07-17 22:07:40.454132737 +0000 UTC m=+3036.434503214"
Jul 17 22:07:40.559421 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.559386    2546 patch_prober.go:28] interesting pod/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Jul 17 22:07:40.559549 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:40.559448    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Jul 17 22:07:41.440309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.440276    2546 generic.go:356] "Generic (PLEG): container finished" podID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerID="66f7d2a0bd2c0b61ff482afef6c63373f7d2af67a074f962af91231225e3934d" exitCode=6
Jul 17 22:07:41.440798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.440366    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerDied","Data":"66f7d2a0bd2c0b61ff482afef6c63373f7d2af67a074f962af91231225e3934d"}
Jul 17 22:07:41.440798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.440527    2546 scope.go:117] "RemoveContainer" containerID="66f7d2a0bd2c0b61ff482afef6c63373f7d2af67a074f962af91231225e3934d"
Jul 17 22:07:41.448505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.442960    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerID="4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2" exitCode=6
Jul 17 22:07:41.448505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.442996    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerDied","Data":"4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2"}
Jul 17 22:07:41.448505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.443085    2546 scope.go:117] "RemoveContainer" containerID="17203933a0c0f46cf364d52ad24c86a92e4da2a2314e0cfc1c1284b47f051679"
Jul 17 22:07:41.448505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.443886    2546 scope.go:117] "RemoveContainer" containerID="4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2"
Jul 17 22:07:41.448505 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:41.444297    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85_odh-ai-gateway-infra(8ee9fc33-9add-4dda-85ef-20534fe40b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45"
Jul 17 22:07:41.566806 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.561225    2546 patch_prober.go:28] interesting pod/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused" start-of-body=
Jul 17 22:07:41.566806 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:41.561291    2546 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.143:15021/healthz/ready\": dial tcp 10.133.0.143:15021: connect: connection refused"
Jul 17 22:07:42.449541 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.449517    2546 scope.go:117] "RemoveContainer" containerID="4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2"
Jul 17 22:07:42.449915 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:42.449754    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85_odh-ai-gateway-infra(8ee9fc33-9add-4dda-85ef-20534fe40b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45"
Jul 17 22:07:42.451109 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.451086    2546 generic.go:356] "Generic (PLEG): container finished" podID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerID="d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd" exitCode=6
Jul 17 22:07:42.451216 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.451169    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerDied","Data":"d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd"}
Jul 17 22:07:42.451311 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.451234    2546 scope.go:117] "RemoveContainer" containerID="66f7d2a0bd2c0b61ff482afef6c63373f7d2af67a074f962af91231225e3934d"
Jul 17 22:07:42.451311 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.451292    2546 scope.go:117] "RemoveContainer" containerID="d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd"
Jul 17 22:07:42.451485 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:42.451466    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj_odh-ai-gateway-infra(72ea0f90-bd32-4029-8294-48cfd9b15fa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6"
Jul 17 22:07:42.562295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.562269    2546 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:42.562665 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.562647    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:42.563218 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:42.563200    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:07:43.458258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:43.458222    2546 scope.go:117] "RemoveContainer" containerID="d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd"
Jul 17 22:07:43.458751 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:43.458468    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj_odh-ai-gateway-infra(72ea0f90-bd32-4029-8294-48cfd9b15fa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6"
Jul 17 22:07:55.530914 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:55.530837    2546 scope.go:117] "RemoveContainer" containerID="4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2"
Jul 17 22:07:56.534153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:56.534116    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerID="029d662ff078c11af3bc2569602c24c7d7538bdb6c283c2430c775be3b477d78" exitCode=6
Jul 17 22:07:56.536335 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:56.536303    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerDied","Data":"029d662ff078c11af3bc2569602c24c7d7538bdb6c283c2430c775be3b477d78"}
Jul 17 22:07:56.536486 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:56.536376    2546 scope.go:117] "RemoveContainer" containerID="4a41a0dd432fccac7b7aa7fad434bbad8bb79c2caecddb3f1ac363ce4cfca4a2"
Jul 17 22:07:56.536715 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:56.536699    2546 scope.go:117] "RemoveContainer" containerID="029d662ff078c11af3bc2569602c24c7d7538bdb6c283c2430c775be3b477d78"
Jul 17 22:07:56.536926 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:56.536907    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85_odh-ai-gateway-infra(8ee9fc33-9add-4dda-85ef-20534fe40b45)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45"
Jul 17 22:07:57.531578 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.531543    2546 scope.go:117] "RemoveContainer" containerID="d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd"
Jul 17 22:07:57.562648 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.562619    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:57.699419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.699394    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:57.801696 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.801634    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca\") pod \"8ee9fc33-9add-4dda-85ef-20534fe40b45\" (UID: \"8ee9fc33-9add-4dda-85ef-20534fe40b45\") "
Jul 17 22:07:57.801983 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.801961    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ee9fc33-9add-4dda-85ef-20534fe40b45" (UID: "8ee9fc33-9add-4dda-85ef-20534fe40b45"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:57.902948 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:57.902926    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee9fc33-9add-4dda-85ef-20534fe40b45-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:07:58.549231 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.549207    2546 generic.go:356] "Generic (PLEG): container finished" podID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerID="74b614e9b2e4dd517b0d27d0ec3e2561930265218b550937550d827bdbf1973e" exitCode=6
Jul 17 22:07:58.549357 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.549275    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerDied","Data":"74b614e9b2e4dd517b0d27d0ec3e2561930265218b550937550d827bdbf1973e"}
Jul 17 22:07:58.549357 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.549334    2546 scope.go:117] "RemoveContainer" containerID="d5500676ecc18bc2b9d216b075e814195622b3c722323bde841d42385240d2bd"
Jul 17 22:07:58.549645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.549629    2546 scope.go:117] "RemoveContainer" containerID="74b614e9b2e4dd517b0d27d0ec3e2561930265218b550937550d827bdbf1973e"
Jul 17 22:07:58.549859 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:07:58.549825    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj_odh-ai-gateway-infra(72ea0f90-bd32-4029-8294-48cfd9b15fa6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6"
Jul 17 22:07:58.550827 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.550802    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85" event={"ID":"8ee9fc33-9add-4dda-85ef-20534fe40b45","Type":"ContainerDied","Data":"ebd158154ed28c9287144acbc188aa59b6e091aa5db04c5b3da7036b6e2c171c"}
Jul 17 22:07:58.550886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.550836    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"
Jul 17 22:07:58.559778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.559763    2546 scope.go:117] "RemoveContainer" containerID="029d662ff078c11af3bc2569602c24c7d7538bdb6c283c2430c775be3b477d78"
Jul 17 22:07:58.580783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.580759    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:58.586735 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:58.586699    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-jgs85"]
Jul 17 22:07:59.578043 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:59.578009    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:07:59.704960 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:59.704939    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:07:59.817391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:59.817354    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca\") pod \"72ea0f90-bd32-4029-8294-48cfd9b15fa6\" (UID: \"72ea0f90-bd32-4029-8294-48cfd9b15fa6\") "
Jul 17 22:07:59.817704 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:59.817685    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72ea0f90-bd32-4029-8294-48cfd9b15fa6" (UID: "72ea0f90-bd32-4029-8294-48cfd9b15fa6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:07:59.918639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:07:59.918569    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72ea0f90-bd32-4029-8294-48cfd9b15fa6-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:08:00.542582 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.542548    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" path="/var/lib/kubelet/pods/8ee9fc33-9add-4dda-85ef-20534fe40b45/volumes"
Jul 17 22:08:00.566282 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.566263    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"
Jul 17 22:08:00.566282 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.566269    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj" event={"ID":"72ea0f90-bd32-4029-8294-48cfd9b15fa6","Type":"ContainerDied","Data":"d424cd47115be9b23b4ca8ed7ff7a1ccaceb95ac39c6cb7e48efdc8027057791"}
Jul 17 22:08:00.566446 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.566328    2546 scope.go:117] "RemoveContainer" containerID="74b614e9b2e4dd517b0d27d0ec3e2561930265218b550937550d827bdbf1973e"
Jul 17 22:08:00.583509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.583487    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:08:00.589716 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:00.589695    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-md2tj"]
Jul 17 22:08:01.013711 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.013675    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.013987    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.013999    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014008    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014014    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014020    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014027    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014034    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014039    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014046    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014082 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014051    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014118    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014125    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014130    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014135    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014140    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee9fc33-9add-4dda-85ef-20534fe40b45" containerName="revoke-keys"
Jul 17 22:08:01.014694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.014145    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:01.016882 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.016865    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:01.019127 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.019104    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:08:01.025136 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.025116    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:01.127690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.127665    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c\" (UID: \"544130c7-6436-4e31-9379-f2d263f0ab58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:01.228895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.228870    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c\" (UID: \"544130c7-6436-4e31-9379-f2d263f0ab58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:01.229513 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.229493    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c\" (UID: \"544130c7-6436-4e31-9379-f2d263f0ab58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:01.328324 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.328241    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:01.459152 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.459129    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:01.461040 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:08:01.461009    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod544130c7_6436_4e31_9379_f2d263f0ab58.slice/crio-71e96d7ccc9ff22026f04d83e7cd02e0c3144333e0ba8a8adc09ae01979ce614 WatchSource:0}: Error finding container 71e96d7ccc9ff22026f04d83e7cd02e0c3144333e0ba8a8adc09ae01979ce614: Status 404 returned error can't find the container with id 71e96d7ccc9ff22026f04d83e7cd02e0c3144333e0ba8a8adc09ae01979ce614
Jul 17 22:08:01.572850 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:01.572812    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" event={"ID":"544130c7-6436-4e31-9379-f2d263f0ab58","Type":"ContainerStarted","Data":"71e96d7ccc9ff22026f04d83e7cd02e0c3144333e0ba8a8adc09ae01979ce614"}
Jul 17 22:08:02.536648 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:02.536621    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" path="/var/lib/kubelet/pods/72ea0f90-bd32-4029-8294-48cfd9b15fa6/volumes"
Jul 17 22:08:02.579834 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:02.579805    2546 generic.go:356] "Generic (PLEG): container finished" podID="544130c7-6436-4e31-9379-f2d263f0ab58" containerID="b50e840faf5d8f8eb56c02e2757d4a24577cb2ac77217ec486ef5bedc2845746" exitCode=6
Jul 17 22:08:02.579967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:02.579891    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" event={"ID":"544130c7-6436-4e31-9379-f2d263f0ab58","Type":"ContainerDied","Data":"b50e840faf5d8f8eb56c02e2757d4a24577cb2ac77217ec486ef5bedc2845746"}
Jul 17 22:08:02.580050 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:02.580032    2546 scope.go:117] "RemoveContainer" containerID="b50e840faf5d8f8eb56c02e2757d4a24577cb2ac77217ec486ef5bedc2845746"
Jul 17 22:08:03.587258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:03.587227    2546 generic.go:356] "Generic (PLEG): container finished" podID="544130c7-6436-4e31-9379-f2d263f0ab58" containerID="a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81" exitCode=6
Jul 17 22:08:03.587698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:03.587283    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" event={"ID":"544130c7-6436-4e31-9379-f2d263f0ab58","Type":"ContainerDied","Data":"a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81"}
Jul 17 22:08:03.587698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:03.587339    2546 scope.go:117] "RemoveContainer" containerID="b50e840faf5d8f8eb56c02e2757d4a24577cb2ac77217ec486ef5bedc2845746"
Jul 17 22:08:03.587698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:03.587490    2546 scope.go:117] "RemoveContainer" containerID="a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81"
Jul 17 22:08:03.587893 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:03.587738    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c_odh-ai-gateway-infra(544130c7-6436-4e31-9379-f2d263f0ab58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" podUID="544130c7-6436-4e31-9379-f2d263f0ab58"
Jul 17 22:08:04.594878 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:04.594850    2546 scope.go:117] "RemoveContainer" containerID="a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81"
Jul 17 22:08:04.595254 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:04.594993    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c_odh-ai-gateway-infra(544130c7-6436-4e31-9379-f2d263f0ab58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" podUID="544130c7-6436-4e31-9379-f2d263f0ab58"
Jul 17 22:08:05.819929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.819896    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:05.820317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.820206    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:05.820317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.820218    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="72ea0f90-bd32-4029-8294-48cfd9b15fa6" containerName="revoke-keys"
Jul 17 22:08:05.823389 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.823372    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:05.836223 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.836191    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:05.966407 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:05.966379    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm\" (UID: \"c964caf6-7eaf-47b1-b31c-2dd511d990a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:06.067410 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.067382    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm\" (UID: \"c964caf6-7eaf-47b1-b31c-2dd511d990a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:06.068053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.068035    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm\" (UID: \"c964caf6-7eaf-47b1-b31c-2dd511d990a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:06.133547 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.133494    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:06.264009 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.263987    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:06.265347 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:08:06.265325    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc964caf6_7eaf_47b1_b31c_2dd511d990a2.slice/crio-cde67aa17b578ab59cb4bae8731281a45ca4038482fc0b6aeb1ee7e4cff870e5 WatchSource:0}: Error finding container cde67aa17b578ab59cb4bae8731281a45ca4038482fc0b6aeb1ee7e4cff870e5: Status 404 returned error can't find the container with id cde67aa17b578ab59cb4bae8731281a45ca4038482fc0b6aeb1ee7e4cff870e5
Jul 17 22:08:06.606951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.606919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerStarted","Data":"1fdd438fe00bc0e80ab3cd46bfa8680393ce5423640efb0541979f2608ed2a73"}
Jul 17 22:08:06.607084 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.606965    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerStarted","Data":"cde67aa17b578ab59cb4bae8731281a45ca4038482fc0b6aeb1ee7e4cff870e5"}
Jul 17 22:08:06.623764 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:06.623703    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" podStartSLOduration=1.6236884900000002 podStartE2EDuration="1.62368849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:06.621429363 +0000 UTC m=+3062.601799840" watchObservedRunningTime="2026-07-17 22:08:06.62368849 +0000 UTC m=+3062.604058988"
Jul 17 22:08:07.613845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:07.613813    2546 generic.go:356] "Generic (PLEG): container finished" podID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerID="1fdd438fe00bc0e80ab3cd46bfa8680393ce5423640efb0541979f2608ed2a73" exitCode=6
Jul 17 22:08:07.614226 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:07.613870    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerDied","Data":"1fdd438fe00bc0e80ab3cd46bfa8680393ce5423640efb0541979f2608ed2a73"}
Jul 17 22:08:07.614226 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:07.614060    2546 scope.go:117] "RemoveContainer" containerID="1fdd438fe00bc0e80ab3cd46bfa8680393ce5423640efb0541979f2608ed2a73"
Jul 17 22:08:08.620680 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:08.620650    2546 generic.go:356] "Generic (PLEG): container finished" podID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerID="1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc" exitCode=6
Jul 17 22:08:08.621106 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:08.620760    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerDied","Data":"1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc"}
Jul 17 22:08:08.621106 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:08.620820    2546 scope.go:117] "RemoveContainer" containerID="1fdd438fe00bc0e80ab3cd46bfa8680393ce5423640efb0541979f2608ed2a73"
Jul 17 22:08:08.621106 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:08.620893    2546 scope.go:117] "RemoveContainer" containerID="1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc"
Jul 17 22:08:08.621277 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:08.621124    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm_odh-ai-gateway-infra(c964caf6-7eaf-47b1-b31c-2dd511d990a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2"
Jul 17 22:08:09.633516 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:09.633491    2546 scope.go:117] "RemoveContainer" containerID="1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc"
Jul 17 22:08:09.633919 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:09.633667    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm_odh-ai-gateway-infra(c964caf6-7eaf-47b1-b31c-2dd511d990a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2"
Jul 17 22:08:19.531147 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:19.531108    2546 scope.go:117] "RemoveContainer" containerID="a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81"
Jul 17 22:08:20.694610 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:20.694577    2546 generic.go:356] "Generic (PLEG): container finished" podID="544130c7-6436-4e31-9379-f2d263f0ab58" containerID="c3b1507eada12953659fb2630fdf4ed9f06abacb43a914bebec74e29aa2b247d" exitCode=6
Jul 17 22:08:20.695135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:20.694637    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" event={"ID":"544130c7-6436-4e31-9379-f2d263f0ab58","Type":"ContainerDied","Data":"c3b1507eada12953659fb2630fdf4ed9f06abacb43a914bebec74e29aa2b247d"}
Jul 17 22:08:20.695135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:20.694698    2546 scope.go:117] "RemoveContainer" containerID="a4afde8f28a60cb1018e863aeb1f4c8768a1535941bd8663ff013e9b532f1f81"
Jul 17 22:08:20.695135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:20.694968    2546 scope.go:117] "RemoveContainer" containerID="c3b1507eada12953659fb2630fdf4ed9f06abacb43a914bebec74e29aa2b247d"
Jul 17 22:08:20.695280 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:20.695173    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c_odh-ai-gateway-infra(544130c7-6436-4e31-9379-f2d263f0ab58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" podUID="544130c7-6436-4e31-9379-f2d263f0ab58"
Jul 17 22:08:21.530875 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:21.530843    2546 scope.go:117] "RemoveContainer" containerID="1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc"
Jul 17 22:08:21.721485 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:21.721454    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:21.856968 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:21.856949    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:21.994374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:21.994344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca\") pod \"544130c7-6436-4e31-9379-f2d263f0ab58\" (UID: \"544130c7-6436-4e31-9379-f2d263f0ab58\") "
Jul 17 22:08:21.994709 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:21.994684    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "544130c7-6436-4e31-9379-f2d263f0ab58" (UID: "544130c7-6436-4e31-9379-f2d263f0ab58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:22.095214 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.095125    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/544130c7-6436-4e31-9379-f2d263f0ab58-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:08:22.706373 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.706347    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"
Jul 17 22:08:22.706552 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.706382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c" event={"ID":"544130c7-6436-4e31-9379-f2d263f0ab58","Type":"ContainerDied","Data":"71e96d7ccc9ff22026f04d83e7cd02e0c3144333e0ba8a8adc09ae01979ce614"}
Jul 17 22:08:22.706552 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.706446    2546 scope.go:117] "RemoveContainer" containerID="c3b1507eada12953659fb2630fdf4ed9f06abacb43a914bebec74e29aa2b247d"
Jul 17 22:08:22.708632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.708589    2546 generic.go:356] "Generic (PLEG): container finished" podID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerID="5491f424f5c9a6fdda9cfc056c51080c22fc544ec0806a5c99ceda00cf6c8811" exitCode=6
Jul 17 22:08:22.708814 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.708662    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerDied","Data":"5491f424f5c9a6fdda9cfc056c51080c22fc544ec0806a5c99ceda00cf6c8811"}
Jul 17 22:08:22.709006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.708990    2546 scope.go:117] "RemoveContainer" containerID="5491f424f5c9a6fdda9cfc056c51080c22fc544ec0806a5c99ceda00cf6c8811"
Jul 17 22:08:22.709194 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:22.709173    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm_odh-ai-gateway-infra(c964caf6-7eaf-47b1-b31c-2dd511d990a2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2"
Jul 17 22:08:22.714990 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.714973    2546 scope.go:117] "RemoveContainer" containerID="1d0969d4520d7b478af51109c363f3b22601711d8b6b34fa1782aa968fd421cc"
Jul 17 22:08:22.723009 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.722989    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:22.730156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:22.729421    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-d8x2c"]
Jul 17 22:08:23.753230 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:23.753197    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:23.883097 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:23.883077    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:24.010140 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.010076    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca\") pod \"c964caf6-7eaf-47b1-b31c-2dd511d990a2\" (UID: \"c964caf6-7eaf-47b1-b31c-2dd511d990a2\") "
Jul 17 22:08:24.010444 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.010424    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c964caf6-7eaf-47b1-b31c-2dd511d990a2" (UID: "c964caf6-7eaf-47b1-b31c-2dd511d990a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:24.111602 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.111562    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c964caf6-7eaf-47b1-b31c-2dd511d990a2-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:08:24.537287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.537259    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" path="/var/lib/kubelet/pods/544130c7-6436-4e31-9379-f2d263f0ab58/volumes"
Jul 17 22:08:24.721695 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.721666    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"
Jul 17 22:08:24.721695 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.721676    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm" event={"ID":"c964caf6-7eaf-47b1-b31c-2dd511d990a2","Type":"ContainerDied","Data":"cde67aa17b578ab59cb4bae8731281a45ca4038482fc0b6aeb1ee7e4cff870e5"}
Jul 17 22:08:24.721945 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.721765    2546 scope.go:117] "RemoveContainer" containerID="5491f424f5c9a6fdda9cfc056c51080c22fc544ec0806a5c99ceda00cf6c8811"
Jul 17 22:08:24.738546 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.738526    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:24.742523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:24.742500    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-2vhzm"]
Jul 17 22:08:26.537051 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:26.537009    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" path="/var/lib/kubelet/pods/c964caf6-7eaf-47b1-b31c-2dd511d990a2/volumes"
Jul 17 22:08:38.397333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397289    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397636    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397648    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397660    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397666    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397675    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397682    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397694    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397699    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397770    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397781    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397787    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397794    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397799    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.397865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.397805    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:38.400690 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.400669    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:38.402994 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.402973    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:08:38.407469 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.407422    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:08:38.426511 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.426482    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx\" (UID: \"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:38.526779 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.526742    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx\" (UID: \"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:38.527299 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.527281    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx\" (UID: \"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:38.712632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.712560    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:38.869310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:38.869286    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:08:38.871518 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:08:38.871491    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1f3d0c8_1f84_43a4_a0ac_ec45b5940ea1.slice/crio-7f3f668d9961548c605d8b589269f0fc605005426e0dd5d2a98a8413fb5156c7 WatchSource:0}: Error finding container 7f3f668d9961548c605d8b589269f0fc605005426e0dd5d2a98a8413fb5156c7: Status 404 returned error can't find the container with id 7f3f668d9961548c605d8b589269f0fc605005426e0dd5d2a98a8413fb5156c7
Jul 17 22:08:39.236394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.236369    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:39.236670 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.236659    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:39.236713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.236671    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="544130c7-6436-4e31-9379-f2d263f0ab58" containerName="revoke-keys"
Jul 17 22:08:39.236713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.236692    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:39.236713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.236697    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c964caf6-7eaf-47b1-b31c-2dd511d990a2" containerName="revoke-keys"
Jul 17 22:08:39.238841 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.238825    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:39.248323 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.248301    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:39.333208 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.333162    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb\" (UID: \"8d438e45-d86b-4dbb-841a-ca15ec715bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:39.433626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.433600    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb\" (UID: \"8d438e45-d86b-4dbb-841a-ca15ec715bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:39.434173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.434155    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb\" (UID: \"8d438e45-d86b-4dbb-841a-ca15ec715bce\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:39.549074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.549046    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:39.676601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.676582    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:39.678529 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:08:39.678506    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d438e45_d86b_4dbb_841a_ca15ec715bce.slice/crio-4063d8657bca5ded96fd2a2911b0b1ff18194f31914ccfcd8e5a93d144e4d9c2 WatchSource:0}: Error finding container 4063d8657bca5ded96fd2a2911b0b1ff18194f31914ccfcd8e5a93d144e4d9c2: Status 404 returned error can't find the container with id 4063d8657bca5ded96fd2a2911b0b1ff18194f31914ccfcd8e5a93d144e4d9c2
Jul 17 22:08:39.809369 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.809341    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerID="83a21d1eb71064e76f5931ccc6c66956ab6c9f7e00cf427684fe38fab4e3bdd3" exitCode=6
Jul 17 22:08:39.809468 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.809402    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerDied","Data":"83a21d1eb71064e76f5931ccc6c66956ab6c9f7e00cf427684fe38fab4e3bdd3"}
Jul 17 22:08:39.809468 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.809452    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerStarted","Data":"7f3f668d9961548c605d8b589269f0fc605005426e0dd5d2a98a8413fb5156c7"}
Jul 17 22:08:39.809656 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.809642    2546 scope.go:117] "RemoveContainer" containerID="83a21d1eb71064e76f5931ccc6c66956ab6c9f7e00cf427684fe38fab4e3bdd3"
Jul 17 22:08:39.810629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:39.810603    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerStarted","Data":"4063d8657bca5ded96fd2a2911b0b1ff18194f31914ccfcd8e5a93d144e4d9c2"}
Jul 17 22:08:40.817841 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.817804    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerID="a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a" exitCode=6
Jul 17 22:08:40.818283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.817883    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerDied","Data":"a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a"}
Jul 17 22:08:40.818283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.817934    2546 scope.go:117] "RemoveContainer" containerID="83a21d1eb71064e76f5931ccc6c66956ab6c9f7e00cf427684fe38fab4e3bdd3"
Jul 17 22:08:40.818283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.818111    2546 scope.go:117] "RemoveContainer" containerID="a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a"
Jul 17 22:08:40.818424 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:40.818297    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx_odh-ai-gateway-infra(c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1"
Jul 17 22:08:40.819534 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.819515    2546 generic.go:356] "Generic (PLEG): container finished" podID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerID="d4035f2a3e0fd5829e91c2c10431d54607162e0996400a42e5bf1da1b2413b8f" exitCode=6
Jul 17 22:08:40.819669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.819609    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerDied","Data":"d4035f2a3e0fd5829e91c2c10431d54607162e0996400a42e5bf1da1b2413b8f"}
Jul 17 22:08:40.819793 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:40.819777    2546 scope.go:117] "RemoveContainer" containerID="d4035f2a3e0fd5829e91c2c10431d54607162e0996400a42e5bf1da1b2413b8f"
Jul 17 22:08:41.827162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:41.827127    2546 scope.go:117] "RemoveContainer" containerID="a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a"
Jul 17 22:08:41.827611 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:41.827350    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx_odh-ai-gateway-infra(c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1"
Jul 17 22:08:41.829004 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:41.828975    2546 generic.go:356] "Generic (PLEG): container finished" podID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerID="c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729" exitCode=6
Jul 17 22:08:41.829128 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:41.829030    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerDied","Data":"c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729"}
Jul 17 22:08:41.829128 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:41.829086    2546 scope.go:117] "RemoveContainer" containerID="d4035f2a3e0fd5829e91c2c10431d54607162e0996400a42e5bf1da1b2413b8f"
Jul 17 22:08:41.829233 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:41.829218    2546 scope.go:117] "RemoveContainer" containerID="c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729"
Jul 17 22:08:41.829402 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:41.829378    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb_odh-ai-gateway-infra(8d438e45-d86b-4dbb-841a-ca15ec715bce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce"
Jul 17 22:08:42.835995 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:42.835968    2546 scope.go:117] "RemoveContainer" containerID="c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729"
Jul 17 22:08:42.836358 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:42.836156    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb_odh-ai-gateway-infra(8d438e45-d86b-4dbb-841a-ca15ec715bce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce"
Jul 17 22:08:54.534341 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:54.534308    2546 scope.go:117] "RemoveContainer" containerID="c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729"
Jul 17 22:08:54.912347 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:54.912316    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerStarted","Data":"7aa15d51661b6494b435107c2cf857946279e30c0061124c8fde79a93ec49d9b"}
Jul 17 22:08:54.928816 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:54.928775    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" podStartSLOduration=15.928761603 podStartE2EDuration="15.928761603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:54.92786924 +0000 UTC m=+3110.908239728" watchObservedRunningTime="2026-07-17 22:08:54.928761603 +0000 UTC m=+3110.909132105"
Jul 17 22:08:55.847835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.847788    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:08:55.851376 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.851349    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:08:55.858649 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.858616    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:08:55.919484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.919455    2546 generic.go:356] "Generic (PLEG): container finished" podID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerID="7aa15d51661b6494b435107c2cf857946279e30c0061124c8fde79a93ec49d9b" exitCode=6
Jul 17 22:08:55.919647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.919538    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerDied","Data":"7aa15d51661b6494b435107c2cf857946279e30c0061124c8fde79a93ec49d9b"}
Jul 17 22:08:55.919647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.919620    2546 scope.go:117] "RemoveContainer" containerID="c91ff2e3c574d70e443208b060bf5b235cdecec579069b9ab96dcd4f730d9729"
Jul 17 22:08:55.919948 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.919928    2546 scope.go:117] "RemoveContainer" containerID="7aa15d51661b6494b435107c2cf857946279e30c0061124c8fde79a93ec49d9b"
Jul 17 22:08:55.920202 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:55.920179    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb_odh-ai-gateway-infra(8d438e45-d86b-4dbb-841a-ca15ec715bce)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce"
Jul 17 22:08:55.941330 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.941307    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:55.966770 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:55.966747    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp\" (UID: \"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:08:56.068193 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.068162    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp\" (UID: \"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:08:56.068713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.068697    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp\" (UID: \"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:08:56.165073 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.164994    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:08:56.297926 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.297899    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:08:56.298928 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:08:56.298903    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cdf68d9_0e9a_4631_815f_6c79ffc9fd58.slice/crio-429e5619ab18f5d24c3cf0269babb411601e7be99b8689c500b342b8e3006a08 WatchSource:0}: Error finding container 429e5619ab18f5d24c3cf0269babb411601e7be99b8689c500b342b8e3006a08: Status 404 returned error can't find the container with id 429e5619ab18f5d24c3cf0269babb411601e7be99b8689c500b342b8e3006a08
Jul 17 22:08:56.531909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.531888    2546 scope.go:117] "RemoveContainer" containerID="a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a"
Jul 17 22:08:56.926546 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.926514    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerStarted","Data":"a6de2563919fcb2d7a195c3c67e3ea425c4e4d479367f4afedf026e4f8a17850"}
Jul 17 22:08:56.929647 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.929625    2546 generic.go:356] "Generic (PLEG): container finished" podID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerID="24bad943f0e7832cca81e527a05eb7704fef1b4a650256517ba1250a02ce275a" exitCode=6
Jul 17 22:08:56.929745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.929707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerDied","Data":"24bad943f0e7832cca81e527a05eb7704fef1b4a650256517ba1250a02ce275a"}
Jul 17 22:08:56.929789 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.929769    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerStarted","Data":"429e5619ab18f5d24c3cf0269babb411601e7be99b8689c500b342b8e3006a08"}
Jul 17 22:08:56.929946 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.929934    2546 scope.go:117] "RemoveContainer" containerID="24bad943f0e7832cca81e527a05eb7704fef1b4a650256517ba1250a02ce275a"
Jul 17 22:08:56.943455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:56.943415    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" podStartSLOduration=18.943402075 podStartE2EDuration="18.943402075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:08:56.940965945 +0000 UTC m=+3112.921336422" watchObservedRunningTime="2026-07-17 22:08:56.943402075 +0000 UTC m=+3112.923772552"
Jul 17 22:08:57.049996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.049972    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:57.078121 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.078092    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca\") pod \"8d438e45-d86b-4dbb-841a-ca15ec715bce\" (UID: \"8d438e45-d86b-4dbb-841a-ca15ec715bce\") "
Jul 17 22:08:57.078403 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.078382    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8d438e45-d86b-4dbb-841a-ca15ec715bce" (UID: "8d438e45-d86b-4dbb-841a-ca15ec715bce"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:57.178639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.178616    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8d438e45-d86b-4dbb-841a-ca15ec715bce-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:08:57.937248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.937185    2546 generic.go:356] "Generic (PLEG): container finished" podID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerID="fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2" exitCode=6
Jul 17 22:08:57.937626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.937264    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerDied","Data":"fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2"}
Jul 17 22:08:57.937626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.937315    2546 scope.go:117] "RemoveContainer" containerID="24bad943f0e7832cca81e527a05eb7704fef1b4a650256517ba1250a02ce275a"
Jul 17 22:08:57.937626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.937450    2546 scope.go:117] "RemoveContainer" containerID="fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2"
Jul 17 22:08:57.937842 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:57.937646    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp_odh-ai-gateway-infra(5cdf68d9-0e9a-4631-815f-6c79ffc9fd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58"
Jul 17 22:08:57.942047 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.941370    2546 generic.go:356] "Generic (PLEG): container finished" podID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerID="a6de2563919fcb2d7a195c3c67e3ea425c4e4d479367f4afedf026e4f8a17850" exitCode=6
Jul 17 22:08:57.942047 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.941460    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerDied","Data":"a6de2563919fcb2d7a195c3c67e3ea425c4e4d479367f4afedf026e4f8a17850"}
Jul 17 22:08:57.942219 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.942169    2546 scope.go:117] "RemoveContainer" containerID="a6de2563919fcb2d7a195c3c67e3ea425c4e4d479367f4afedf026e4f8a17850"
Jul 17 22:08:57.942368 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:57.942343    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx_odh-ai-gateway-infra(c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1"
Jul 17 22:08:57.943933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.943900    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb" event={"ID":"8d438e45-d86b-4dbb-841a-ca15ec715bce","Type":"ContainerDied","Data":"4063d8657bca5ded96fd2a2911b0b1ff18194f31914ccfcd8e5a93d144e4d9c2"}
Jul 17 22:08:57.944025 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.943969    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"
Jul 17 22:08:57.949948 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.949924    2546 scope.go:117] "RemoveContainer" containerID="a50edd549a3204642ca36d572fdc3175904f1c605afe263b18ed628008cd3e0a"
Jul 17 22:08:57.951907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.951888    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:08:57.961850 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.961834    2546 scope.go:117] "RemoveContainer" containerID="7aa15d51661b6494b435107c2cf857946279e30c0061124c8fde79a93ec49d9b"
Jul 17 22:08:57.994145 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.994128    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:57.998113 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:57.998091    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7vbhb"]
Jul 17 22:08:58.537219 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:58.537182    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" path="/var/lib/kubelet/pods/8d438e45-d86b-4dbb-841a-ca15ec715bce/volumes"
Jul 17 22:08:58.955212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:58.955185    2546 scope.go:117] "RemoveContainer" containerID="fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2"
Jul 17 22:08:58.955584 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:08:58.955367    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp_odh-ai-gateway-infra(5cdf68d9-0e9a-4631-815f-6c79ffc9fd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58"
Jul 17 22:08:59.095802 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.095767    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:59.198551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.198528    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca\") pod \"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1\" (UID: \"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1\") "
Jul 17 22:08:59.198860 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.198837    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" (UID: "c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:08:59.301327 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.301252    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:08:59.962068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.962039    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"
Jul 17 22:08:59.962541 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.962070    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx" event={"ID":"c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1","Type":"ContainerDied","Data":"7f3f668d9961548c605d8b589269f0fc605005426e0dd5d2a98a8413fb5156c7"}
Jul 17 22:08:59.962541 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.962133    2546 scope.go:117] "RemoveContainer" containerID="a6de2563919fcb2d7a195c3c67e3ea425c4e4d479367f4afedf026e4f8a17850"
Jul 17 22:08:59.988048 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.988025    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:08:59.994113 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:08:59.994093    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-w2whx"]
Jul 17 22:09:00.537376 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:00.537340    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" path="/var/lib/kubelet/pods/c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1/volumes"
Jul 17 22:09:01.056673 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.056642    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057001    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057013    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057032    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057039    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057056    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057062    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057068    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057074    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057120    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057128    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057134    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.057317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057139    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:01.057317 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.057145    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:01.061603 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.061585    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:01.068996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.068913    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:01.116139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.116112    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p\" (UID: \"5ebd7815-dbce-4ed0-960b-f44cce8cf868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:01.217181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.217155    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p\" (UID: \"5ebd7815-dbce-4ed0-960b-f44cce8cf868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:01.217683 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.217666    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p\" (UID: \"5ebd7815-dbce-4ed0-960b-f44cce8cf868\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:01.372693 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.372633    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:01.504645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.504621    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:01.506552 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:09:01.506530    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ebd7815_dbce_4ed0_960b_f44cce8cf868.slice/crio-095b6666fdca07430a116d31a971b6c3d254dd16f97f7d382165e11ee8b736fb WatchSource:0}: Error finding container 095b6666fdca07430a116d31a971b6c3d254dd16f97f7d382165e11ee8b736fb: Status 404 returned error can't find the container with id 095b6666fdca07430a116d31a971b6c3d254dd16f97f7d382165e11ee8b736fb
Jul 17 22:09:01.975700 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.975612    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerStarted","Data":"5d8160d283f4d1e86714c75686a9cedb01332ade6da317b883586f49086238da"}
Jul 17 22:09:01.975700 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.975676    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerStarted","Data":"095b6666fdca07430a116d31a971b6c3d254dd16f97f7d382165e11ee8b736fb"}
Jul 17 22:09:01.994409 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:01.994360    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" podStartSLOduration=0.994341247 podStartE2EDuration="994.341247ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:01.993555404 +0000 UTC m=+3117.973925891" watchObservedRunningTime="2026-07-17 22:09:01.994341247 +0000 UTC m=+3117.974711726"
Jul 17 22:09:02.982401 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:02.982364    2546 generic.go:356] "Generic (PLEG): container finished" podID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerID="5d8160d283f4d1e86714c75686a9cedb01332ade6da317b883586f49086238da" exitCode=6
Jul 17 22:09:02.982821 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:02.982456    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerDied","Data":"5d8160d283f4d1e86714c75686a9cedb01332ade6da317b883586f49086238da"}
Jul 17 22:09:02.982821 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:02.982604    2546 scope.go:117] "RemoveContainer" containerID="5d8160d283f4d1e86714c75686a9cedb01332ade6da317b883586f49086238da"
Jul 17 22:09:03.989197 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:03.989167    2546 generic.go:356] "Generic (PLEG): container finished" podID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerID="fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8" exitCode=6
Jul 17 22:09:03.989652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:03.989225    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerDied","Data":"fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8"}
Jul 17 22:09:03.989652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:03.989272    2546 scope.go:117] "RemoveContainer" containerID="5d8160d283f4d1e86714c75686a9cedb01332ade6da317b883586f49086238da"
Jul 17 22:09:03.989652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:03.989419    2546 scope.go:117] "RemoveContainer" containerID="fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8"
Jul 17 22:09:03.989652 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:03.989613    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p_odh-ai-gateway-infra(5ebd7815-dbce-4ed0-960b-f44cce8cf868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868"
Jul 17 22:09:05.000302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:05.000276    2546 scope.go:117] "RemoveContainer" containerID="fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8"
Jul 17 22:09:05.000679 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:05.000450    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p_odh-ai-gateway-infra(5ebd7815-dbce-4ed0-960b-f44cce8cf868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868"
Jul 17 22:09:11.530983 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:11.530946    2546 scope.go:117] "RemoveContainer" containerID="fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2"
Jul 17 22:09:12.038741 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:12.038690    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerStarted","Data":"7f111b590505c2ea6cd33a3c88a74104dbaa8531f5883af8e1f3894f66a97971"}
Jul 17 22:09:12.055249 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:12.055213    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" podStartSLOduration=17.05520028 podStartE2EDuration="17.05520028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:08:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:12.054490993 +0000 UTC m=+3128.034861492" watchObservedRunningTime="2026-07-17 22:09:12.05520028 +0000 UTC m=+3128.035570757"
Jul 17 22:09:13.046355 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:13.046314    2546 generic.go:356] "Generic (PLEG): container finished" podID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerID="7f111b590505c2ea6cd33a3c88a74104dbaa8531f5883af8e1f3894f66a97971" exitCode=6
Jul 17 22:09:13.046868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:13.046385    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerDied","Data":"7f111b590505c2ea6cd33a3c88a74104dbaa8531f5883af8e1f3894f66a97971"}
Jul 17 22:09:13.046868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:13.046449    2546 scope.go:117] "RemoveContainer" containerID="fd0ed6ff25fb86c0d1a7e3b9bfa4fae619db3ebc641c68dd930b537df1cc77b2"
Jul 17 22:09:13.046868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:13.046757    2546 scope.go:117] "RemoveContainer" containerID="7f111b590505c2ea6cd33a3c88a74104dbaa8531f5883af8e1f3894f66a97971"
Jul 17 22:09:13.047055 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:13.046973    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp_odh-ai-gateway-infra(5cdf68d9-0e9a-4631-815f-6c79ffc9fd58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58"
Jul 17 22:09:13.065219 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:13.065197    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:09:14.180804 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:14.180778    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:09:14.315775 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:14.315672    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca\") pod \"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58\" (UID: \"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58\") "
Jul 17 22:09:14.316007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:14.315987    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" (UID: "5cdf68d9-0e9a-4631-815f-6c79ffc9fd58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:14.416285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:14.416249    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:09:15.060002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:15.059969    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"
Jul 17 22:09:15.060002 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:15.059978    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp" event={"ID":"5cdf68d9-0e9a-4631-815f-6c79ffc9fd58","Type":"ContainerDied","Data":"429e5619ab18f5d24c3cf0269babb411601e7be99b8689c500b342b8e3006a08"}
Jul 17 22:09:15.060249 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:15.060067    2546 scope.go:117] "RemoveContainer" containerID="7f111b590505c2ea6cd33a3c88a74104dbaa8531f5883af8e1f3894f66a97971"
Jul 17 22:09:15.077374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:15.077351    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:09:15.083266 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:15.083217    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pnxfp"]
Jul 17 22:09:16.537190 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:16.537143    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" path="/var/lib/kubelet/pods/5cdf68d9-0e9a-4631-815f-6c79ffc9fd58/volumes"
Jul 17 22:09:18.531597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:18.531560    2546 scope.go:117] "RemoveContainer" containerID="fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8"
Jul 17 22:09:19.082555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:19.082477    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerStarted","Data":"6141b77e0f0ca105b6e8c6878ccfcd8dd190ad4ae71cb7c35d35c7449f602640"}
Jul 17 22:09:20.089126 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:20.089093    2546 generic.go:356] "Generic (PLEG): container finished" podID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerID="6141b77e0f0ca105b6e8c6878ccfcd8dd190ad4ae71cb7c35d35c7449f602640" exitCode=6
Jul 17 22:09:20.089548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:20.089166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerDied","Data":"6141b77e0f0ca105b6e8c6878ccfcd8dd190ad4ae71cb7c35d35c7449f602640"}
Jul 17 22:09:20.089548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:20.089226    2546 scope.go:117] "RemoveContainer" containerID="fac64204379f30fb48663278088775f39444e982f0816c210f965e5c0a3d07c8"
Jul 17 22:09:20.089548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:20.089472    2546 scope.go:117] "RemoveContainer" containerID="6141b77e0f0ca105b6e8c6878ccfcd8dd190ad4ae71cb7c35d35c7449f602640"
Jul 17 22:09:20.089700 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:20.089678    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p_odh-ai-gateway-infra(5ebd7815-dbce-4ed0-960b-f44cce8cf868)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868"
Jul 17 22:09:20.114078 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:20.114057    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:21.223792 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:21.223769    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:21.372511 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:21.372449    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca\") pod \"5ebd7815-dbce-4ed0-960b-f44cce8cf868\" (UID: \"5ebd7815-dbce-4ed0-960b-f44cce8cf868\") "
Jul 17 22:09:21.372784 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:21.372761    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5ebd7815-dbce-4ed0-960b-f44cce8cf868" (UID: "5ebd7815-dbce-4ed0-960b-f44cce8cf868"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:21.473259 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:21.473234    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5ebd7815-dbce-4ed0-960b-f44cce8cf868-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:09:22.103028 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.103002    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"
Jul 17 22:09:22.103220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.103006    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p" event={"ID":"5ebd7815-dbce-4ed0-960b-f44cce8cf868","Type":"ContainerDied","Data":"095b6666fdca07430a116d31a971b6c3d254dd16f97f7d382165e11ee8b736fb"}
Jul 17 22:09:22.103220 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.103131    2546 scope.go:117] "RemoveContainer" containerID="6141b77e0f0ca105b6e8c6878ccfcd8dd190ad4ae71cb7c35d35c7449f602640"
Jul 17 22:09:22.126886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.126856    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:22.130588 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.130566    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dbb7p"]
Jul 17 22:09:22.536301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:22.536267    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" path="/var/lib/kubelet/pods/5ebd7815-dbce-4ed0-960b-f44cce8cf868/volumes"
Jul 17 22:09:29.284809 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.284771    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285082    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285094    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285105    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285110    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285121    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285130    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d438e45-d86b-4dbb-841a-ca15ec715bce" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285137    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285142    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285152    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285157    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285169    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285175    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285182    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285187    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285239    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285247    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285253    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285260    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1f3d0c8-1f84-43a4-a0ac-ec45b5940ea1" containerName="revoke-keys"
Jul 17 22:09:29.285555 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.285266    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:29.289388 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.289372    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:29.291668 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.291647    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:09:29.295325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.295301    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:29.438388 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.438359    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk\" (UID: \"e3974dbc-1fa1-4f03-94e6-061a23374775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:29.539120 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.539056    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk\" (UID: \"e3974dbc-1fa1-4f03-94e6-061a23374775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:29.539641 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.539621    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk\" (UID: \"e3974dbc-1fa1-4f03-94e6-061a23374775\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:29.601683 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.601662    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:29.739930 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:29.739902    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:29.741808 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:09:29.741782    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3974dbc_1fa1_4f03_94e6_061a23374775.slice/crio-b6e321a99ff8642d1a612733979e732bdc20da922d6e9cc382bd89bd9f67d713 WatchSource:0}: Error finding container b6e321a99ff8642d1a612733979e732bdc20da922d6e9cc382bd89bd9f67d713: Status 404 returned error can't find the container with id b6e321a99ff8642d1a612733979e732bdc20da922d6e9cc382bd89bd9f67d713
Jul 17 22:09:30.146280 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:30.146243    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerStarted","Data":"16625bdaf5288a9c2b9fa44317f767df03beb90521ff70cfc065584bd26cb9fc"}
Jul 17 22:09:30.146436 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:30.146300    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerStarted","Data":"b6e321a99ff8642d1a612733979e732bdc20da922d6e9cc382bd89bd9f67d713"}
Jul 17 22:09:30.162632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:30.162587    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" podStartSLOduration=1.162570807 podStartE2EDuration="1.162570807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:09:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:09:30.161910361 +0000 UTC m=+3146.142280840" watchObservedRunningTime="2026-07-17 22:09:30.162570807 +0000 UTC m=+3146.142941288"
Jul 17 22:09:31.152387 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:31.152350    2546 generic.go:356] "Generic (PLEG): container finished" podID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerID="16625bdaf5288a9c2b9fa44317f767df03beb90521ff70cfc065584bd26cb9fc" exitCode=6
Jul 17 22:09:31.152865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:31.152447    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerDied","Data":"16625bdaf5288a9c2b9fa44317f767df03beb90521ff70cfc065584bd26cb9fc"}
Jul 17 22:09:31.152865 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:31.152611    2546 scope.go:117] "RemoveContainer" containerID="16625bdaf5288a9c2b9fa44317f767df03beb90521ff70cfc065584bd26cb9fc"
Jul 17 22:09:32.159957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:32.159865    2546 generic.go:356] "Generic (PLEG): container finished" podID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerID="6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd" exitCode=6
Jul 17 22:09:32.159957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:32.159934    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerDied","Data":"6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd"}
Jul 17 22:09:32.160469 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:32.160007    2546 scope.go:117] "RemoveContainer" containerID="16625bdaf5288a9c2b9fa44317f767df03beb90521ff70cfc065584bd26cb9fc"
Jul 17 22:09:32.160469 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:32.160248    2546 scope.go:117] "RemoveContainer" containerID="6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd"
Jul 17 22:09:32.160469 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:32.160464    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk_odh-ai-gateway-infra(e3974dbc-1fa1-4f03-94e6-061a23374775)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775"
Jul 17 22:09:33.167271 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:33.167245    2546 scope.go:117] "RemoveContainer" containerID="6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd"
Jul 17 22:09:33.167662 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:33.167400    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk_odh-ai-gateway-infra(e3974dbc-1fa1-4f03-94e6-061a23374775)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775"
Jul 17 22:09:38.440908 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.440875    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:38.441295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.441249    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:38.441295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.441262    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:38.441371 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.441327    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5ebd7815-dbce-4ed0-960b-f44cce8cf868" containerName="revoke-keys"
Jul 17 22:09:38.441371 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.441340    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cdf68d9-0e9a-4631-815f-6c79ffc9fd58" containerName="revoke-keys"
Jul 17 22:09:38.445463 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.445448    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:38.451156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.451132    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:38.514537 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.514511    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm\" (UID: \"18024b72-1c9b-4c20-8bbd-0c00220da49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:38.615373 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.615344    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm\" (UID: \"18024b72-1c9b-4c20-8bbd-0c00220da49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:38.615967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.615949    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm\" (UID: \"18024b72-1c9b-4c20-8bbd-0c00220da49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:38.756630 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:38.756605    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:39.096122 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:39.096093    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:39.098037 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:09:39.097997    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18024b72_1c9b_4c20_8bbd_0c00220da49c.slice/crio-99398616ac107effa15472cd3f8193f88db1380d47d0a8fedf750e4b81464c88 WatchSource:0}: Error finding container 99398616ac107effa15472cd3f8193f88db1380d47d0a8fedf750e4b81464c88: Status 404 returned error can't find the container with id 99398616ac107effa15472cd3f8193f88db1380d47d0a8fedf750e4b81464c88
Jul 17 22:09:39.199466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:39.199438    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" event={"ID":"18024b72-1c9b-4c20-8bbd-0c00220da49c","Type":"ContainerStarted","Data":"99398616ac107effa15472cd3f8193f88db1380d47d0a8fedf750e4b81464c88"}
Jul 17 22:09:40.206296 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:40.206259    2546 generic.go:356] "Generic (PLEG): container finished" podID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerID="6ef6556756fce8271f54efe28d818ba70079d4ec2dc0310dd1e4c9b58488adc2" exitCode=6
Jul 17 22:09:40.206648 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:40.206341    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" event={"ID":"18024b72-1c9b-4c20-8bbd-0c00220da49c","Type":"ContainerDied","Data":"6ef6556756fce8271f54efe28d818ba70079d4ec2dc0310dd1e4c9b58488adc2"}
Jul 17 22:09:40.206648 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:40.206481    2546 scope.go:117] "RemoveContainer" containerID="6ef6556756fce8271f54efe28d818ba70079d4ec2dc0310dd1e4c9b58488adc2"
Jul 17 22:09:41.212773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:41.212698    2546 generic.go:356] "Generic (PLEG): container finished" podID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerID="41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793" exitCode=6
Jul 17 22:09:41.213156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:41.212770    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" event={"ID":"18024b72-1c9b-4c20-8bbd-0c00220da49c","Type":"ContainerDied","Data":"41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793"}
Jul 17 22:09:41.213156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:41.212823    2546 scope.go:117] "RemoveContainer" containerID="6ef6556756fce8271f54efe28d818ba70079d4ec2dc0310dd1e4c9b58488adc2"
Jul 17 22:09:41.213156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:41.212987    2546 scope.go:117] "RemoveContainer" containerID="41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793"
Jul 17 22:09:41.213280 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:41.213186    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm_odh-ai-gateway-infra(18024b72-1c9b-4c20-8bbd-0c00220da49c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c"
Jul 17 22:09:42.220573 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:42.220544    2546 scope.go:117] "RemoveContainer" containerID="41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793"
Jul 17 22:09:42.221066 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:42.220761    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm_odh-ai-gateway-infra(18024b72-1c9b-4c20-8bbd-0c00220da49c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c"
Jul 17 22:09:45.888867 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:45.888827    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:09:45.892564 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:45.892535    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:09:45.900426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:45.900396    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:09:45.968705 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:45.968673    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-298mf\" (UID: \"05e919a2-e368-47eb-86e1-e75b2ffeec95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:09:46.069560 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:46.069528    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-298mf\" (UID: \"05e919a2-e368-47eb-86e1-e75b2ffeec95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:09:46.070149 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:46.070129    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-298mf\" (UID: \"05e919a2-e368-47eb-86e1-e75b2ffeec95\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:09:46.205929 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:46.205859    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:09:46.336391 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:46.336366    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:09:46.338900 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:09:46.338876    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05e919a2_e368_47eb_86e1_e75b2ffeec95.slice/crio-37d91463e93eb0d1387b6a5693ec71e4bdc5a97056fbcad600d192ec315ecd4f WatchSource:0}: Error finding container 37d91463e93eb0d1387b6a5693ec71e4bdc5a97056fbcad600d192ec315ecd4f: Status 404 returned error can't find the container with id 37d91463e93eb0d1387b6a5693ec71e4bdc5a97056fbcad600d192ec315ecd4f
Jul 17 22:09:47.249941 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:47.249899    2546 generic.go:356] "Generic (PLEG): container finished" podID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerID="6d5c85dd47b2217bc3705d01b28afccb831230f4649aff2850fef95c0bc69f8b" exitCode=6
Jul 17 22:09:47.250310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:47.249976    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" event={"ID":"05e919a2-e368-47eb-86e1-e75b2ffeec95","Type":"ContainerDied","Data":"6d5c85dd47b2217bc3705d01b28afccb831230f4649aff2850fef95c0bc69f8b"}
Jul 17 22:09:47.250310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:47.250024    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" event={"ID":"05e919a2-e368-47eb-86e1-e75b2ffeec95","Type":"ContainerStarted","Data":"37d91463e93eb0d1387b6a5693ec71e4bdc5a97056fbcad600d192ec315ecd4f"}
Jul 17 22:09:47.250310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:47.250216    2546 scope.go:117] "RemoveContainer" containerID="6d5c85dd47b2217bc3705d01b28afccb831230f4649aff2850fef95c0bc69f8b"
Jul 17 22:09:48.256692 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:48.256658    2546 generic.go:356] "Generic (PLEG): container finished" podID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerID="5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536" exitCode=6
Jul 17 22:09:48.257156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:48.256757    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" event={"ID":"05e919a2-e368-47eb-86e1-e75b2ffeec95","Type":"ContainerDied","Data":"5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536"}
Jul 17 22:09:48.257156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:48.256851    2546 scope.go:117] "RemoveContainer" containerID="6d5c85dd47b2217bc3705d01b28afccb831230f4649aff2850fef95c0bc69f8b"
Jul 17 22:09:48.257156 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:48.257041    2546 scope.go:117] "RemoveContainer" containerID="5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536"
Jul 17 22:09:48.257297 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:48.257276    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-298mf_odh-ai-gateway-infra(05e919a2-e368-47eb-86e1-e75b2ffeec95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95"
Jul 17 22:09:48.531125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:48.531070    2546 scope.go:117] "RemoveContainer" containerID="6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd"
Jul 17 22:09:49.263787 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:49.263758    2546 generic.go:356] "Generic (PLEG): container finished" podID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerID="192d5efc612c95ce19fe4071e5b06468c338fa20a9c05456463b05dab0c23c5d" exitCode=6
Jul 17 22:09:49.264207 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:49.263833    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerDied","Data":"192d5efc612c95ce19fe4071e5b06468c338fa20a9c05456463b05dab0c23c5d"}
Jul 17 22:09:49.264207 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:49.263899    2546 scope.go:117] "RemoveContainer" containerID="6c751706b48557ff289fdd84f4beda9dc1c2ca7cee6af518570ec6fd49d4b6dd"
Jul 17 22:09:49.264207 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:49.264144    2546 scope.go:117] "RemoveContainer" containerID="192d5efc612c95ce19fe4071e5b06468c338fa20a9c05456463b05dab0c23c5d"
Jul 17 22:09:49.264406 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:49.264381    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk_odh-ai-gateway-infra(e3974dbc-1fa1-4f03-94e6-061a23374775)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775"
Jul 17 22:09:49.265788 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:49.265772    2546 scope.go:117] "RemoveContainer" containerID="5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536"
Jul 17 22:09:49.265939 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:49.265924    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-298mf_odh-ai-gateway-infra(05e919a2-e368-47eb-86e1-e75b2ffeec95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95"
Jul 17 22:09:50.291636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:50.291599    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:50.418238 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:50.418217    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:50.502889 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:50.502858    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca\") pod \"e3974dbc-1fa1-4f03-94e6-061a23374775\" (UID: \"e3974dbc-1fa1-4f03-94e6-061a23374775\") "
Jul 17 22:09:50.503158 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:50.503138    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3974dbc-1fa1-4f03-94e6-061a23374775" (UID: "e3974dbc-1fa1-4f03-94e6-061a23374775"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:50.604020 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:50.603995    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3974dbc-1fa1-4f03-94e6-061a23374775-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:09:51.278756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:51.278707    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk" event={"ID":"e3974dbc-1fa1-4f03-94e6-061a23374775","Type":"ContainerDied","Data":"b6e321a99ff8642d1a612733979e732bdc20da922d6e9cc382bd89bd9f67d713"}
Jul 17 22:09:51.278756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:51.278750    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"
Jul 17 22:09:51.278996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:51.278790    2546 scope.go:117] "RemoveContainer" containerID="192d5efc612c95ce19fe4071e5b06468c338fa20a9c05456463b05dab0c23c5d"
Jul 17 22:09:51.297964 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:51.297936    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:51.303569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:51.303548    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-cxndk"]
Jul 17 22:09:52.536068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:52.536035    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" path="/var/lib/kubelet/pods/e3974dbc-1fa1-4f03-94e6-061a23374775/volumes"
Jul 17 22:09:55.531478 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:55.531451    2546 scope.go:117] "RemoveContainer" containerID="41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793"
Jul 17 22:09:56.307752 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:56.307701    2546 generic.go:356] "Generic (PLEG): container finished" podID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerID="9a81ddcf9ca589443c81fdb7b7abe9d85275434e2f2e2a74249b9a56378fb841" exitCode=6
Jul 17 22:09:56.307963 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:56.307750    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" event={"ID":"18024b72-1c9b-4c20-8bbd-0c00220da49c","Type":"ContainerDied","Data":"9a81ddcf9ca589443c81fdb7b7abe9d85275434e2f2e2a74249b9a56378fb841"}
Jul 17 22:09:56.307963 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:56.307819    2546 scope.go:117] "RemoveContainer" containerID="41a796e585f6ccd178927e4295625f6b2b29162cd358455f3c0bfa437797c793"
Jul 17 22:09:56.308092 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:56.308075    2546 scope.go:117] "RemoveContainer" containerID="9a81ddcf9ca589443c81fdb7b7abe9d85275434e2f2e2a74249b9a56378fb841"
Jul 17 22:09:56.308290 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:09:56.308273    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm_odh-ai-gateway-infra(18024b72-1c9b-4c20-8bbd-0c00220da49c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c"
Jul 17 22:09:57.379090 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:57.379050    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:57.518467 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:57.518439    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:57.560644 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:57.560601    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca\") pod \"18024b72-1c9b-4c20-8bbd-0c00220da49c\" (UID: \"18024b72-1c9b-4c20-8bbd-0c00220da49c\") "
Jul 17 22:09:57.561004 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:57.560981    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "18024b72-1c9b-4c20-8bbd-0c00220da49c" (UID: "18024b72-1c9b-4c20-8bbd-0c00220da49c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:09:57.661825 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:57.661694    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/18024b72-1c9b-4c20-8bbd-0c00220da49c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:09:58.321008 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.320977    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"
Jul 17 22:09:58.321193 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.320968    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm" event={"ID":"18024b72-1c9b-4c20-8bbd-0c00220da49c","Type":"ContainerDied","Data":"99398616ac107effa15472cd3f8193f88db1380d47d0a8fedf750e4b81464c88"}
Jul 17 22:09:58.321193 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.321116    2546 scope.go:117] "RemoveContainer" containerID="9a81ddcf9ca589443c81fdb7b7abe9d85275434e2f2e2a74249b9a56378fb841"
Jul 17 22:09:58.342797 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.342774    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:58.347122 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.347099    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-p65fm"]
Jul 17 22:09:58.542045 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:09:58.542017    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" path="/var/lib/kubelet/pods/18024b72-1c9b-4c20-8bbd-0c00220da49c/volumes"
Jul 17 22:10:01.107742 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.107693    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108030    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108043    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108054    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108059    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108071    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108076    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108083    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108088    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108095    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108100    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108114    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108139 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108119    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108166    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108174    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108181    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.108502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108186    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:01.108502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.108191    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3974dbc-1fa1-4f03-94e6-061a23374775" containerName="revoke-keys"
Jul 17 22:10:01.112805 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.112788    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:01.118053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.118028    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:01.190096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.190069    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh\" (UID: \"a052db07-b602-4dc6-ab6c-5370d5b70a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:01.291232 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.291205    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh\" (UID: \"a052db07-b602-4dc6-ab6c-5370d5b70a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:01.291888 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.291869    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh\" (UID: \"a052db07-b602-4dc6-ab6c-5370d5b70a58\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:01.425473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.425416    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:01.765570 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:01.765547    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:01.766251 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:10:01.766224    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda052db07_b602_4dc6_ab6c_5370d5b70a58.slice/crio-807ef201244f76daf3cd4494128c050e307e5f881778762f79de78886184d478 WatchSource:0}: Error finding container 807ef201244f76daf3cd4494128c050e307e5f881778762f79de78886184d478: Status 404 returned error can't find the container with id 807ef201244f76daf3cd4494128c050e307e5f881778762f79de78886184d478
Jul 17 22:10:02.353269 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:02.353169    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerStarted","Data":"9c5aecc91e0a6c76d47c820b9c5928c6c383e0dc2727f3f5bbf09428f256d3a1"}
Jul 17 22:10:02.353269 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:02.353222    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerStarted","Data":"807ef201244f76daf3cd4494128c050e307e5f881778762f79de78886184d478"}
Jul 17 22:10:02.368125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:02.368067    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" podStartSLOduration=1.368046533 podStartE2EDuration="1.368046533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:02.367275251 +0000 UTC m=+3178.347645749" watchObservedRunningTime="2026-07-17 22:10:02.368046533 +0000 UTC m=+3178.348417011"
Jul 17 22:10:03.360012 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:03.359975    2546 generic.go:356] "Generic (PLEG): container finished" podID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerID="9c5aecc91e0a6c76d47c820b9c5928c6c383e0dc2727f3f5bbf09428f256d3a1" exitCode=6
Jul 17 22:10:03.360012 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:03.360014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerDied","Data":"9c5aecc91e0a6c76d47c820b9c5928c6c383e0dc2727f3f5bbf09428f256d3a1"}
Jul 17 22:10:03.360573 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:03.360249    2546 scope.go:117] "RemoveContainer" containerID="9c5aecc91e0a6c76d47c820b9c5928c6c383e0dc2727f3f5bbf09428f256d3a1"
Jul 17 22:10:04.366710 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:04.366676    2546 generic.go:356] "Generic (PLEG): container finished" podID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerID="731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48" exitCode=6
Jul 17 22:10:04.367196 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:04.366744    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerDied","Data":"731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48"}
Jul 17 22:10:04.367196 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:04.366794    2546 scope.go:117] "RemoveContainer" containerID="9c5aecc91e0a6c76d47c820b9c5928c6c383e0dc2727f3f5bbf09428f256d3a1"
Jul 17 22:10:04.367196 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:04.367033    2546 scope.go:117] "RemoveContainer" containerID="731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48"
Jul 17 22:10:04.367376 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:04.367249    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh_odh-ai-gateway-infra(a052db07-b602-4dc6-ab6c-5370d5b70a58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58"
Jul 17 22:10:04.534521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:04.534496    2546 scope.go:117] "RemoveContainer" containerID="5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536"
Jul 17 22:10:05.373574 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:05.373541    2546 generic.go:356] "Generic (PLEG): container finished" podID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerID="7bbcae149cea3906e67e442808fff6421052acae7a329110eb8b090ff0ca0cfc" exitCode=6
Jul 17 22:10:05.374007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:05.373615    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" event={"ID":"05e919a2-e368-47eb-86e1-e75b2ffeec95","Type":"ContainerDied","Data":"7bbcae149cea3906e67e442808fff6421052acae7a329110eb8b090ff0ca0cfc"}
Jul 17 22:10:05.374007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:05.373665    2546 scope.go:117] "RemoveContainer" containerID="5cdecfb933cf98e0fac977891c9c4a509eaeb31225e17663dd0f44cfcebc0536"
Jul 17 22:10:05.374007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:05.373995    2546 scope.go:117] "RemoveContainer" containerID="7bbcae149cea3906e67e442808fff6421052acae7a329110eb8b090ff0ca0cfc"
Jul 17 22:10:05.374228 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:05.374207    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-298mf_odh-ai-gateway-infra(05e919a2-e368-47eb-86e1-e75b2ffeec95)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95"
Jul 17 22:10:05.375465 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:05.375451    2546 scope.go:117] "RemoveContainer" containerID="731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48"
Jul 17 22:10:05.381131 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:05.376076    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh_odh-ai-gateway-infra(a052db07-b602-4dc6-ab6c-5370d5b70a58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58"
Jul 17 22:10:06.409393 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:06.409361    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:10:06.546952 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:06.546926    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:10:06.639486 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:06.639461    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca\") pod \"05e919a2-e368-47eb-86e1-e75b2ffeec95\" (UID: \"05e919a2-e368-47eb-86e1-e75b2ffeec95\") "
Jul 17 22:10:06.639807 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:06.639787    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05e919a2-e368-47eb-86e1-e75b2ffeec95" (UID: "05e919a2-e368-47eb-86e1-e75b2ffeec95"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:06.740214 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:06.740189    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05e919a2-e368-47eb-86e1-e75b2ffeec95-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:10:07.388305 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:07.388276    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"
Jul 17 22:10:07.388305 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:07.388293    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf" event={"ID":"05e919a2-e368-47eb-86e1-e75b2ffeec95","Type":"ContainerDied","Data":"37d91463e93eb0d1387b6a5693ec71e4bdc5a97056fbcad600d192ec315ecd4f"}
Jul 17 22:10:07.388548 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:07.388360    2546 scope.go:117] "RemoveContainer" containerID="7bbcae149cea3906e67e442808fff6421052acae7a329110eb8b090ff0ca0cfc"
Jul 17 22:10:07.412338 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:07.412308    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:10:07.416606 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:07.416581    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-298mf"]
Jul 17 22:10:08.536475 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:08.536437    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" path="/var/lib/kubelet/pods/05e919a2-e368-47eb-86e1-e75b2ffeec95/volumes"
Jul 17 22:10:18.531809 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:18.531780    2546 scope.go:117] "RemoveContainer" containerID="731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48"
Jul 17 22:10:19.455664 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:19.455568    2546 generic.go:356] "Generic (PLEG): container finished" podID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerID="842c87bdf1e70ebcebad91573ab888f71a742a67866a6e7c582333cbd90bdc55" exitCode=6
Jul 17 22:10:19.455664 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:19.455639    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerDied","Data":"842c87bdf1e70ebcebad91573ab888f71a742a67866a6e7c582333cbd90bdc55"}
Jul 17 22:10:19.455893 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:19.455698    2546 scope.go:117] "RemoveContainer" containerID="731b7461727f1e7f5b53a75b1ac09ba9b25c8b5445bb8ba2667d150fa09a2d48"
Jul 17 22:10:19.456004 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:19.455976    2546 scope.go:117] "RemoveContainer" containerID="842c87bdf1e70ebcebad91573ab888f71a742a67866a6e7c582333cbd90bdc55"
Jul 17 22:10:19.456213 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:19.456188    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh_odh-ai-gateway-infra(a052db07-b602-4dc6-ab6c-5370d5b70a58)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58"
Jul 17 22:10:20.488403 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:20.488369    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:20.619538 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:20.619512    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:20.744316 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:20.744254    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca\") pod \"a052db07-b602-4dc6-ab6c-5370d5b70a58\" (UID: \"a052db07-b602-4dc6-ab6c-5370d5b70a58\") "
Jul 17 22:10:20.744575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:20.744554    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a052db07-b602-4dc6-ab6c-5370d5b70a58" (UID: "a052db07-b602-4dc6-ab6c-5370d5b70a58"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:20.845187 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:20.845160    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a052db07-b602-4dc6-ab6c-5370d5b70a58-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:10:21.470589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:21.470561    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"
Jul 17 22:10:21.470589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:21.470570    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh" event={"ID":"a052db07-b602-4dc6-ab6c-5370d5b70a58","Type":"ContainerDied","Data":"807ef201244f76daf3cd4494128c050e307e5f881778762f79de78886184d478"}
Jul 17 22:10:21.470870 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:21.470643    2546 scope.go:117] "RemoveContainer" containerID="842c87bdf1e70ebcebad91573ab888f71a742a67866a6e7c582333cbd90bdc55"
Jul 17 22:10:21.491679 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:21.491652    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:21.498132 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:21.498106    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-w62xh"]
Jul 17 22:10:22.536092 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:22.536058    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" path="/var/lib/kubelet/pods/a052db07-b602-4dc6-ab6c-5370d5b70a58/volumes"
Jul 17 22:10:28.479800 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.479763    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480068    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480079    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480104    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480110    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480121    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480127    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480133    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480138    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480144    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480149    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480165    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480170    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480217    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480226    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="18024b72-1c9b-4c20-8bbd-0c00220da49c" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480233    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480239    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480244    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:28.480597 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.480248    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="05e919a2-e368-47eb-86e1-e75b2ffeec95" containerName="revoke-keys"
Jul 17 22:10:28.484524 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.484504    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:28.487208 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.487186    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:10:28.489982 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.489957    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:28.603413 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.603378    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs\" (UID: \"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:28.704650 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.704621    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs\" (UID: \"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:28.705239 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.705217    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs\" (UID: \"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:28.796369 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.796346    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:28.931673 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:28.931648    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:28.933531 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:10:28.933503    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c0d6f6_07b6_43f9_b7a5_579d0058ac1a.slice/crio-e7ad7e322ba7201bdc983a63c52baa5fa982435bf9aa761c3495e1dc4275add9 WatchSource:0}: Error finding container e7ad7e322ba7201bdc983a63c52baa5fa982435bf9aa761c3495e1dc4275add9: Status 404 returned error can't find the container with id e7ad7e322ba7201bdc983a63c52baa5fa982435bf9aa761c3495e1dc4275add9
Jul 17 22:10:29.328264 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.328226    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:29.328652 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.328634    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a052db07-b602-4dc6-ab6c-5370d5b70a58" containerName="revoke-keys"
Jul 17 22:10:29.331684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.331664    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:29.340626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.340605    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:29.409697 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.409670    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7\" (UID: \"a0223d4e-97b1-47c8-8cd5-b690745561ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:29.510049 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.510021    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7\" (UID: \"a0223d4e-97b1-47c8-8cd5-b690745561ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:29.510650 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.510627    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7\" (UID: \"a0223d4e-97b1-47c8-8cd5-b690745561ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:29.515569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.515543    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerStarted","Data":"7a52428df479a610a3ebad71aba66d74418febf66f4c4dda598719c49c40a993"}
Jul 17 22:10:29.515661 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.515587    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerStarted","Data":"e7ad7e322ba7201bdc983a63c52baa5fa982435bf9aa761c3495e1dc4275add9"}
Jul 17 22:10:29.532688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.532636    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" podStartSLOduration=1.5326199790000001 podStartE2EDuration="1.532619979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:29.531091972 +0000 UTC m=+3205.511462449" watchObservedRunningTime="2026-07-17 22:10:29.532619979 +0000 UTC m=+3205.512990457"
Jul 17 22:10:29.642871 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.642852    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:29.767987 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:29.767965    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:29.769405 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:10:29.769379    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0223d4e_97b1_47c8_8cd5_b690745561ca.slice/crio-c8e65c6518b65c2c6acaafec2f819c6d9ac237fd37466a4515eee3da9b9acc42 WatchSource:0}: Error finding container c8e65c6518b65c2c6acaafec2f819c6d9ac237fd37466a4515eee3da9b9acc42: Status 404 returned error can't find the container with id c8e65c6518b65c2c6acaafec2f819c6d9ac237fd37466a4515eee3da9b9acc42
Jul 17 22:10:30.522328 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.522295    2546 generic.go:356] "Generic (PLEG): container finished" podID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerID="7a52428df479a610a3ebad71aba66d74418febf66f4c4dda598719c49c40a993" exitCode=6
Jul 17 22:10:30.522763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.522390    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerDied","Data":"7a52428df479a610a3ebad71aba66d74418febf66f4c4dda598719c49c40a993"}
Jul 17 22:10:30.522763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.522511    2546 scope.go:117] "RemoveContainer" containerID="7a52428df479a610a3ebad71aba66d74418febf66f4c4dda598719c49c40a993"
Jul 17 22:10:30.523957 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.523937    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerID="fa0f70d2e0f3dfdc7b0b04de7f46247c4ffe8c46a5c725dc16e7984bf6f84d31" exitCode=6
Jul 17 22:10:30.524064 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.523986    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" event={"ID":"a0223d4e-97b1-47c8-8cd5-b690745561ca","Type":"ContainerDied","Data":"fa0f70d2e0f3dfdc7b0b04de7f46247c4ffe8c46a5c725dc16e7984bf6f84d31"}
Jul 17 22:10:30.524064 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.524018    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" event={"ID":"a0223d4e-97b1-47c8-8cd5-b690745561ca","Type":"ContainerStarted","Data":"c8e65c6518b65c2c6acaafec2f819c6d9ac237fd37466a4515eee3da9b9acc42"}
Jul 17 22:10:30.524151 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:30.524141    2546 scope.go:117] "RemoveContainer" containerID="fa0f70d2e0f3dfdc7b0b04de7f46247c4ffe8c46a5c725dc16e7984bf6f84d31"
Jul 17 22:10:31.531283 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.531249    2546 generic.go:356] "Generic (PLEG): container finished" podID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerID="21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012" exitCode=6
Jul 17 22:10:31.531771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.531293    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerDied","Data":"21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012"}
Jul 17 22:10:31.531771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.531371    2546 scope.go:117] "RemoveContainer" containerID="7a52428df479a610a3ebad71aba66d74418febf66f4c4dda598719c49c40a993"
Jul 17 22:10:31.531771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.531559    2546 scope.go:117] "RemoveContainer" containerID="21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012"
Jul 17 22:10:31.531931 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:31.531808    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs_odh-ai-gateway-infra(23c0d6f6-07b6-43f9-b7a5-579d0058ac1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a"
Jul 17 22:10:31.533470 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.533445    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerID="88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd" exitCode=6
Jul 17 22:10:31.533598 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.533527    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" event={"ID":"a0223d4e-97b1-47c8-8cd5-b690745561ca","Type":"ContainerDied","Data":"88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd"}
Jul 17 22:10:31.533703 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.533689    2546 scope.go:117] "RemoveContainer" containerID="88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd"
Jul 17 22:10:31.533893 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:31.533874    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7_odh-ai-gateway-infra(a0223d4e-97b1-47c8-8cd5-b690745561ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca"
Jul 17 22:10:31.543417 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:31.543401    2546 scope.go:117] "RemoveContainer" containerID="fa0f70d2e0f3dfdc7b0b04de7f46247c4ffe8c46a5c725dc16e7984bf6f84d31"
Jul 17 22:10:32.539908 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:32.539881    2546 scope.go:117] "RemoveContainer" containerID="88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd"
Jul 17 22:10:32.540282 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:32.540039    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7_odh-ai-gateway-infra(a0223d4e-97b1-47c8-8cd5-b690745561ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca"
Jul 17 22:10:32.541554 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:32.541535    2546 scope.go:117] "RemoveContainer" containerID="21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012"
Jul 17 22:10:32.541688 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:32.541673    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs_odh-ai-gateway-infra(23c0d6f6-07b6-43f9-b7a5-579d0058ac1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a"
Jul 17 22:10:45.531674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:45.531645    2546 scope.go:117] "RemoveContainer" containerID="88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd"
Jul 17 22:10:45.938711 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:45.938686    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:10:45.941933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:45.941914    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:10:45.952615 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:45.952588    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:10:46.036242 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.036178    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx\" (UID: \"e70325b6-72f6-4738-9ba9-eb894f78718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:10:46.136888 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.136864    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx\" (UID: \"e70325b6-72f6-4738-9ba9-eb894f78718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:10:46.137455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.137437    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx\" (UID: \"e70325b6-72f6-4738-9ba9-eb894f78718c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:10:46.253307 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.253288    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:10:46.379095 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.379073    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:10:46.380514 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:10:46.380486    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode70325b6_72f6_4738_9ba9_eb894f78718c.slice/crio-fe2b8553a3ae063990ac57459e980fb5a3ca68c5d795a9bac4011ef51045892d WatchSource:0}: Error finding container fe2b8553a3ae063990ac57459e980fb5a3ca68c5d795a9bac4011ef51045892d: Status 404 returned error can't find the container with id fe2b8553a3ae063990ac57459e980fb5a3ca68c5d795a9bac4011ef51045892d
Jul 17 22:10:46.531627 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.531602    2546 scope.go:117] "RemoveContainer" containerID="21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012"
Jul 17 22:10:46.617856 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.617834    2546 generic.go:356] "Generic (PLEG): container finished" podID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerID="1dfe669e2c49508889f5195a41f95d0f99306c33162853092f8a0c1b33e2e7c9" exitCode=6
Jul 17 22:10:46.618209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.617910    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" event={"ID":"a0223d4e-97b1-47c8-8cd5-b690745561ca","Type":"ContainerDied","Data":"1dfe669e2c49508889f5195a41f95d0f99306c33162853092f8a0c1b33e2e7c9"}
Jul 17 22:10:46.618209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.617966    2546 scope.go:117] "RemoveContainer" containerID="88a2cb069fd4ab7bda0e1e63edfbaaea0099e072929aa9893754a0524fc5a7bd"
Jul 17 22:10:46.618313 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.618256    2546 scope.go:117] "RemoveContainer" containerID="1dfe669e2c49508889f5195a41f95d0f99306c33162853092f8a0c1b33e2e7c9"
Jul 17 22:10:46.618463 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:46.618439    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7_odh-ai-gateway-infra(a0223d4e-97b1-47c8-8cd5-b690745561ca)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca"
Jul 17 22:10:46.619775 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.619755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerStarted","Data":"a8d017045d870eae924ac08f5bdc04c9cf930bb0c983084b640ac6f3920ea457"}
Jul 17 22:10:46.619864 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.619800    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerStarted","Data":"fe2b8553a3ae063990ac57459e980fb5a3ca68c5d795a9bac4011ef51045892d"}
Jul 17 22:10:46.649499 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:46.649457    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" podStartSLOduration=1.649443912 podStartE2EDuration="1.649443912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:10:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:10:46.645736268 +0000 UTC m=+3222.626106738" watchObservedRunningTime="2026-07-17 22:10:46.649443912 +0000 UTC m=+3222.629814389"
Jul 17 22:10:47.627211 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.627172    2546 generic.go:356] "Generic (PLEG): container finished" podID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerID="a8d017045d870eae924ac08f5bdc04c9cf930bb0c983084b640ac6f3920ea457" exitCode=6
Jul 17 22:10:47.627640 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.627251    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerDied","Data":"a8d017045d870eae924ac08f5bdc04c9cf930bb0c983084b640ac6f3920ea457"}
Jul 17 22:10:47.627640 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.627423    2546 scope.go:117] "RemoveContainer" containerID="a8d017045d870eae924ac08f5bdc04c9cf930bb0c983084b640ac6f3920ea457"
Jul 17 22:10:47.628995 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.628976    2546 generic.go:356] "Generic (PLEG): container finished" podID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerID="4f4efa270cf6368fb660a2de6eae6b39c7ccee4a35ecf8cae9afea9e6591f72e" exitCode=6
Jul 17 22:10:47.629085 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.629046    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerDied","Data":"4f4efa270cf6368fb660a2de6eae6b39c7ccee4a35ecf8cae9afea9e6591f72e"}
Jul 17 22:10:47.629124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.629095    2546 scope.go:117] "RemoveContainer" containerID="21c1c91b33628b09f974a195748212016608abb411d472b5fcee0e036c004012"
Jul 17 22:10:47.629323 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.629311    2546 scope.go:117] "RemoveContainer" containerID="4f4efa270cf6368fb660a2de6eae6b39c7ccee4a35ecf8cae9afea9e6591f72e"
Jul 17 22:10:47.629488 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:47.629471    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs_odh-ai-gateway-infra(23c0d6f6-07b6-43f9-b7a5-579d0058ac1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a"
Jul 17 22:10:47.643460 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.643433    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:47.794344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.794322    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:47.948993 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.948935    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca\") pod \"a0223d4e-97b1-47c8-8cd5-b690745561ca\" (UID: \"a0223d4e-97b1-47c8-8cd5-b690745561ca\") "
Jul 17 22:10:47.949242 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:47.949221    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0223d4e-97b1-47c8-8cd5-b690745561ca" (UID: "a0223d4e-97b1-47c8-8cd5-b690745561ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:48.050185 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.050165    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0223d4e-97b1-47c8-8cd5-b690745561ca-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:10:48.635914 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.635887    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"
Jul 17 22:10:48.636346 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.635930    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7" event={"ID":"a0223d4e-97b1-47c8-8cd5-b690745561ca","Type":"ContainerDied","Data":"c8e65c6518b65c2c6acaafec2f819c6d9ac237fd37466a4515eee3da9b9acc42"}
Jul 17 22:10:48.636346 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.635994    2546 scope.go:117] "RemoveContainer" containerID="1dfe669e2c49508889f5195a41f95d0f99306c33162853092f8a0c1b33e2e7c9"
Jul 17 22:10:48.638114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.638090    2546 generic.go:356] "Generic (PLEG): container finished" podID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerID="11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef" exitCode=6
Jul 17 22:10:48.638257 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.638236    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerDied","Data":"11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef"}
Jul 17 22:10:48.638496 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.638479    2546 scope.go:117] "RemoveContainer" containerID="11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef"
Jul 17 22:10:48.638688 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:48.638669    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx_odh-ai-gateway-infra(e70325b6-72f6-4738-9ba9-eb894f78718c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c"
Jul 17 22:10:48.645212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.645181    2546 scope.go:117] "RemoveContainer" containerID="a8d017045d870eae924ac08f5bdc04c9cf930bb0c983084b640ac6f3920ea457"
Jul 17 22:10:48.670083 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.670056    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:48.673190 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.673170    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:48.675302 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.675280    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-tntp7"]
Jul 17 22:10:48.800484 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.800429    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:48.957463 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.957424    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca\") pod \"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a\" (UID: \"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a\") "
Jul 17 22:10:48.957767 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:48.957747    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" (UID: "23c0d6f6-07b6-43f9-b7a5-579d0058ac1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:10:49.058824 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.058765    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:10:49.648902 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.648874    2546 scope.go:117] "RemoveContainer" containerID="11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef"
Jul 17 22:10:49.649493 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:10:49.649106    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx_odh-ai-gateway-infra(e70325b6-72f6-4738-9ba9-eb894f78718c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c"
Jul 17 22:10:49.649869 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.649852    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"
Jul 17 22:10:49.649967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.649876    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs" event={"ID":"23c0d6f6-07b6-43f9-b7a5-579d0058ac1a","Type":"ContainerDied","Data":"e7ad7e322ba7201bdc983a63c52baa5fa982435bf9aa761c3495e1dc4275add9"}
Jul 17 22:10:49.649967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.649924    2546 scope.go:117] "RemoveContainer" containerID="4f4efa270cf6368fb660a2de6eae6b39c7ccee4a35ecf8cae9afea9e6591f72e"
Jul 17 22:10:49.676321 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.676299    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:49.683783 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:49.683762    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-mtbcs"]
Jul 17 22:10:50.536773 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:50.536737    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" path="/var/lib/kubelet/pods/23c0d6f6-07b6-43f9-b7a5-579d0058ac1a/volumes"
Jul 17 22:10:50.537075 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:10:50.537061    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" path="/var/lib/kubelet/pods/a0223d4e-97b1-47c8-8cd5-b690745561ca/volumes"
Jul 17 22:11:01.150161 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150076    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150442    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150458    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150480    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150486    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150493    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150498    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150514    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150519    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150524    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150589 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150530    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150603    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150613    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150623    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150631    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150638    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.150974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.150644    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23c0d6f6-07b6-43f9-b7a5-579d0058ac1a" containerName="revoke-keys"
Jul 17 22:11:01.155045 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.155028    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:01.160577 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.160552    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:01.252382 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.252356    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6\" (UID: \"e08856cd-99d4-41ac-8a38-e28b44d3adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:01.353192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.353166    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6\" (UID: \"e08856cd-99d4-41ac-8a38-e28b44d3adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:01.353808 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.353791    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6\" (UID: \"e08856cd-99d4-41ac-8a38-e28b44d3adcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:01.466677 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.466609    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:01.598068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.598042    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:01.600071 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:11:01.600043    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode08856cd_99d4_41ac_8a38_e28b44d3adcc.slice/crio-bd52f24306edcaced1429aaffee26d5a4e76acbd8afb02a6031c54d477007d11 WatchSource:0}: Error finding container bd52f24306edcaced1429aaffee26d5a4e76acbd8afb02a6031c54d477007d11: Status 404 returned error can't find the container with id bd52f24306edcaced1429aaffee26d5a4e76acbd8afb02a6031c54d477007d11
Jul 17 22:11:01.716155 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:01.716119    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerStarted","Data":"bd52f24306edcaced1429aaffee26d5a4e76acbd8afb02a6031c54d477007d11"}
Jul 17 22:11:02.723218 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:02.723187    2546 generic.go:356] "Generic (PLEG): container finished" podID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerID="ec0a91eb895b4db9c4e06a67f6a159a865b57e051747434f3e3ad3450a4678b6" exitCode=6
Jul 17 22:11:02.723604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:02.723268    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerDied","Data":"ec0a91eb895b4db9c4e06a67f6a159a865b57e051747434f3e3ad3450a4678b6"}
Jul 17 22:11:02.723604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:02.723402    2546 scope.go:117] "RemoveContainer" containerID="ec0a91eb895b4db9c4e06a67f6a159a865b57e051747434f3e3ad3450a4678b6"
Jul 17 22:11:03.531738 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:03.531697    2546 scope.go:117] "RemoveContainer" containerID="11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef"
Jul 17 22:11:03.730977 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:03.730950    2546 generic.go:356] "Generic (PLEG): container finished" podID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerID="4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac" exitCode=6
Jul 17 22:11:03.731374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:03.731031    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerDied","Data":"4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac"}
Jul 17 22:11:03.731374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:03.731102    2546 scope.go:117] "RemoveContainer" containerID="ec0a91eb895b4db9c4e06a67f6a159a865b57e051747434f3e3ad3450a4678b6"
Jul 17 22:11:03.731374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:03.731276    2546 scope.go:117] "RemoveContainer" containerID="4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac"
Jul 17 22:11:03.731534 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:03.731478    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6_odh-ai-gateway-infra(e08856cd-99d4-41ac-8a38-e28b44d3adcc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc"
Jul 17 22:11:04.738370 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:04.738338    2546 scope.go:117] "RemoveContainer" containerID="4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac"
Jul 17 22:11:04.738777 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:04.738551    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6_odh-ai-gateway-infra(e08856cd-99d4-41ac-8a38-e28b44d3adcc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc"
Jul 17 22:11:04.740186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:04.740162    2546 generic.go:356] "Generic (PLEG): container finished" podID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerID="6ef88c2e63846a114f985eee55733ba2a19420d3ae25519865736e8cdc69365c" exitCode=6
Jul 17 22:11:04.740308 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:04.740204    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerDied","Data":"6ef88c2e63846a114f985eee55733ba2a19420d3ae25519865736e8cdc69365c"}
Jul 17 22:11:04.740308 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:04.740260    2546 scope.go:117] "RemoveContainer" containerID="11fbd22a30884d1f7ab68349a5da1ea7d88a3dbb776fad4192e7f36f632a11ef"
Jul 17 22:11:04.740567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:04.740548    2546 scope.go:117] "RemoveContainer" containerID="6ef88c2e63846a114f985eee55733ba2a19420d3ae25519865736e8cdc69365c"
Jul 17 22:11:04.740764 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:04.740714    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx_odh-ai-gateway-infra(e70325b6-72f6-4738-9ba9-eb894f78718c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c"
Jul 17 22:11:05.778473 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:05.778435    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:11:05.927478 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:05.927453    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:11:06.093203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.093106    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca\") pod \"e70325b6-72f6-4738-9ba9-eb894f78718c\" (UID: \"e70325b6-72f6-4738-9ba9-eb894f78718c\") "
Jul 17 22:11:06.093462 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.093438    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e70325b6-72f6-4738-9ba9-eb894f78718c" (UID: "e70325b6-72f6-4738-9ba9-eb894f78718c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:06.193776 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.193751    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e70325b6-72f6-4738-9ba9-eb894f78718c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:11:06.755492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.755462    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"
Jul 17 22:11:06.755492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.755480    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx" event={"ID":"e70325b6-72f6-4738-9ba9-eb894f78718c","Type":"ContainerDied","Data":"fe2b8553a3ae063990ac57459e980fb5a3ca68c5d795a9bac4011ef51045892d"}
Jul 17 22:11:06.755713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.755539    2546 scope.go:117] "RemoveContainer" containerID="6ef88c2e63846a114f985eee55733ba2a19420d3ae25519865736e8cdc69365c"
Jul 17 22:11:06.772869 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.772840    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:11:06.776368 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:06.776325    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-pttrx"]
Jul 17 22:11:08.536688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:08.536651    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" path="/var/lib/kubelet/pods/e70325b6-72f6-4738-9ba9-eb894f78718c/volumes"
Jul 17 22:11:15.531320 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:15.531288    2546 scope.go:117] "RemoveContainer" containerID="4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac"
Jul 17 22:11:15.809026 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:15.808994    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerStarted","Data":"a474aed3e19c8354d71dcce2b9556a611ca37a21207679545da3078bd00be2ff"}
Jul 17 22:11:15.856981 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:15.856940    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" podStartSLOduration=14.856927023 podStartE2EDuration="14.856927023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:15.855521638 +0000 UTC m=+3251.835892114" watchObservedRunningTime="2026-07-17 22:11:15.856927023 +0000 UTC m=+3251.837297499"
Jul 17 22:11:16.816214 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:16.816182    2546 generic.go:356] "Generic (PLEG): container finished" podID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerID="a474aed3e19c8354d71dcce2b9556a611ca37a21207679545da3078bd00be2ff" exitCode=6
Jul 17 22:11:16.816672 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:16.816256    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerDied","Data":"a474aed3e19c8354d71dcce2b9556a611ca37a21207679545da3078bd00be2ff"}
Jul 17 22:11:16.816672 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:16.816316    2546 scope.go:117] "RemoveContainer" containerID="4d1a703265c5e092bfb303048223a9d80faefd76687cc040c1260a54602ea4ac"
Jul 17 22:11:16.816672 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:16.816558    2546 scope.go:117] "RemoveContainer" containerID="a474aed3e19c8354d71dcce2b9556a611ca37a21207679545da3078bd00be2ff"
Jul 17 22:11:16.816876 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:16.816778    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6_odh-ai-gateway-infra(e08856cd-99d4-41ac-8a38-e28b44d3adcc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc"
Jul 17 22:11:16.867808 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:16.867784    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:17.960600 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:17.960575    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:17.970326 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:17.970306    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca\") pod \"e08856cd-99d4-41ac-8a38-e28b44d3adcc\" (UID: \"e08856cd-99d4-41ac-8a38-e28b44d3adcc\") "
Jul 17 22:11:17.970643 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:17.970622    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e08856cd-99d4-41ac-8a38-e28b44d3adcc" (UID: "e08856cd-99d4-41ac-8a38-e28b44d3adcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:18.071394 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.071363    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e08856cd-99d4-41ac-8a38-e28b44d3adcc-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:11:18.829768 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.829684    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"
Jul 17 22:11:18.829902 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.829679    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6" event={"ID":"e08856cd-99d4-41ac-8a38-e28b44d3adcc","Type":"ContainerDied","Data":"bd52f24306edcaced1429aaffee26d5a4e76acbd8afb02a6031c54d477007d11"}
Jul 17 22:11:18.829902 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.829826    2546 scope.go:117] "RemoveContainer" containerID="a474aed3e19c8354d71dcce2b9556a611ca37a21207679545da3078bd00be2ff"
Jul 17 22:11:18.847243 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.847217    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:18.851641 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:18.851619    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-kcmf6"]
Jul 17 22:11:19.378159 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378129    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378421    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378431    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378444    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378449    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378459    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378465    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378473    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378478    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378488    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:19.378519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378493    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0223d4e-97b1-47c8-8cd5-b690745561ca" containerName="revoke-keys"
Jul 17 22:11:19.378859 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378538    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378859 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378546    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:19.378859 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378552    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.378859 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.378560    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:19.381284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.381268    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:19.383539 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.383521    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:11:19.388471 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.388451    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:19.482116 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.482090    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg\" (UID: \"2eb5cb29-a396-440d-b0eb-f929ef4c30a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:19.583074 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.583050    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg\" (UID: \"2eb5cb29-a396-440d-b0eb-f929ef4c30a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:19.583636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.583616    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg\" (UID: \"2eb5cb29-a396-440d-b0eb-f929ef4c30a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:19.693639 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:19.693569    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:20.028525 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.028498    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:20.030114 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:11:20.030083    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2eb5cb29_a396_440d_b0eb_f929ef4c30a8.slice/crio-0e89cbd05350279731ed97aa9778e70ccdce45965afe1a231206a584a395fc00 WatchSource:0}: Error finding container 0e89cbd05350279731ed97aa9778e70ccdce45965afe1a231206a584a395fc00: Status 404 returned error can't find the container with id 0e89cbd05350279731ed97aa9778e70ccdce45965afe1a231206a584a395fc00
Jul 17 22:11:20.537343 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.537275    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" path="/var/lib/kubelet/pods/e08856cd-99d4-41ac-8a38-e28b44d3adcc/volumes"
Jul 17 22:11:20.845245 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.845163    2546 generic.go:356] "Generic (PLEG): container finished" podID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerID="7ac727b6837c295126c58ffb292bfbdbb5858fe3dbc2fe97fea39d00ebbae430" exitCode=6
Jul 17 22:11:20.845399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.845258    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerDied","Data":"7ac727b6837c295126c58ffb292bfbdbb5858fe3dbc2fe97fea39d00ebbae430"}
Jul 17 22:11:20.845399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.845314    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerStarted","Data":"0e89cbd05350279731ed97aa9778e70ccdce45965afe1a231206a584a395fc00"}
Jul 17 22:11:20.845502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:20.845405    2546 scope.go:117] "RemoveContainer" containerID="7ac727b6837c295126c58ffb292bfbdbb5858fe3dbc2fe97fea39d00ebbae430"
Jul 17 22:11:21.852859 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:21.852830    2546 generic.go:356] "Generic (PLEG): container finished" podID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerID="759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37" exitCode=6
Jul 17 22:11:21.853303 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:21.852927    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerDied","Data":"759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37"}
Jul 17 22:11:21.853303 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:21.852994    2546 scope.go:117] "RemoveContainer" containerID="7ac727b6837c295126c58ffb292bfbdbb5858fe3dbc2fe97fea39d00ebbae430"
Jul 17 22:11:21.853303 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:21.853120    2546 scope.go:117] "RemoveContainer" containerID="759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37"
Jul 17 22:11:21.853482 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:21.853322    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg_odh-ai-gateway-infra(2eb5cb29-a396-440d-b0eb-f929ef4c30a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8"
Jul 17 22:11:22.860601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:22.860574    2546 scope.go:117] "RemoveContainer" containerID="759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37"
Jul 17 22:11:22.861064 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:22.860785    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg_odh-ai-gateway-infra(2eb5cb29-a396-440d-b0eb-f929ef4c30a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8"
Jul 17 22:11:28.526567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526524    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526871    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526884    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526906    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526911    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526962    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e70325b6-72f6-4738-9ba9-eb894f78718c" containerName="revoke-keys"
Jul 17 22:11:28.527000 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.526975    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="e08856cd-99d4-41ac-8a38-e28b44d3adcc" containerName="revoke-keys"
Jul 17 22:11:28.531443 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.531417    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:28.539003 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.538969    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:28.541313 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.541276    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4\" (UID: \"4db6116a-b165-4911-9524-7a9dd3a64efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:28.641758 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.641717    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4\" (UID: \"4db6116a-b165-4911-9524-7a9dd3a64efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:28.642267 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.642249    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4\" (UID: \"4db6116a-b165-4911-9524-7a9dd3a64efb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:28.844423 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.844340    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:28.987132 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:28.987107    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:28.988639 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:11:28.988616    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4db6116a_b165_4911_9524_7a9dd3a64efb.slice/crio-906410ab57b9cbe83dfa15c796c479b917d6719d800ec59ed6970f85d76a07f7 WatchSource:0}: Error finding container 906410ab57b9cbe83dfa15c796c479b917d6719d800ec59ed6970f85d76a07f7: Status 404 returned error can't find the container with id 906410ab57b9cbe83dfa15c796c479b917d6719d800ec59ed6970f85d76a07f7
Jul 17 22:11:29.904832 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:29.904742    2546 generic.go:356] "Generic (PLEG): container finished" podID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerID="27d158a51813ac2146359968a3ecc1dc304471927bf303760978b8102a1ac2f4" exitCode=6
Jul 17 22:11:29.905199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:29.904817    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerDied","Data":"27d158a51813ac2146359968a3ecc1dc304471927bf303760978b8102a1ac2f4"}
Jul 17 22:11:29.905199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:29.904898    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerStarted","Data":"906410ab57b9cbe83dfa15c796c479b917d6719d800ec59ed6970f85d76a07f7"}
Jul 17 22:11:29.905199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:29.904971    2546 scope.go:117] "RemoveContainer" containerID="27d158a51813ac2146359968a3ecc1dc304471927bf303760978b8102a1ac2f4"
Jul 17 22:11:30.912030 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:30.911995    2546 generic.go:356] "Generic (PLEG): container finished" podID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerID="17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c" exitCode=6
Jul 17 22:11:30.912495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:30.912080    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerDied","Data":"17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c"}
Jul 17 22:11:30.912495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:30.912140    2546 scope.go:117] "RemoveContainer" containerID="27d158a51813ac2146359968a3ecc1dc304471927bf303760978b8102a1ac2f4"
Jul 17 22:11:30.912495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:30.912349    2546 scope.go:117] "RemoveContainer" containerID="17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c"
Jul 17 22:11:30.912622 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:30.912566    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4_odh-ai-gateway-infra(4db6116a-b165-4911-9524-7a9dd3a64efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb"
Jul 17 22:11:31.919174 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:31.919139    2546 scope.go:117] "RemoveContainer" containerID="17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c"
Jul 17 22:11:31.919628 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:31.919338    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4_odh-ai-gateway-infra(4db6116a-b165-4911-9524-7a9dd3a64efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb"
Jul 17 22:11:37.531154 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:37.531122    2546 scope.go:117] "RemoveContainer" containerID="759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37"
Jul 17 22:11:37.955604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:37.955565    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerStarted","Data":"aaf18f50d32c27200c2d827f2658fb2f50b632b74dc0368f45bd096462fb4e88"}
Jul 17 22:11:37.971830 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:37.971786    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" podStartSLOduration=18.971770777 podStartE2EDuration="18.971770777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:37.970636658 +0000 UTC m=+3273.951007137" watchObservedRunningTime="2026-07-17 22:11:37.971770777 +0000 UTC m=+3273.952141253"
Jul 17 22:11:38.966173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:38.966140    2546 generic.go:356] "Generic (PLEG): container finished" podID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerID="aaf18f50d32c27200c2d827f2658fb2f50b632b74dc0368f45bd096462fb4e88" exitCode=6
Jul 17 22:11:38.966635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:38.966212    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerDied","Data":"aaf18f50d32c27200c2d827f2658fb2f50b632b74dc0368f45bd096462fb4e88"}
Jul 17 22:11:38.966635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:38.966268    2546 scope.go:117] "RemoveContainer" containerID="759bc2fcd9b659220a1347e5c288c9484316fc76da469662333224b18ad26c37"
Jul 17 22:11:38.966635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:38.966603    2546 scope.go:117] "RemoveContainer" containerID="aaf18f50d32c27200c2d827f2658fb2f50b632b74dc0368f45bd096462fb4e88"
Jul 17 22:11:38.966868 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:38.966844    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg_odh-ai-gateway-infra(2eb5cb29-a396-440d-b0eb-f929ef4c30a8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8"
Jul 17 22:11:38.981216 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:38.981193    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:40.104292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.104269    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:40.120123 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.120099    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca\") pod \"2eb5cb29-a396-440d-b0eb-f929ef4c30a8\" (UID: \"2eb5cb29-a396-440d-b0eb-f929ef4c30a8\") "
Jul 17 22:11:40.120737 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.120453    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2eb5cb29-a396-440d-b0eb-f929ef4c30a8" (UID: "2eb5cb29-a396-440d-b0eb-f929ef4c30a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:40.221284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.221266    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2eb5cb29-a396-440d-b0eb-f929ef4c30a8-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:11:40.979262 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.979184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg" event={"ID":"2eb5cb29-a396-440d-b0eb-f929ef4c30a8","Type":"ContainerDied","Data":"0e89cbd05350279731ed97aa9778e70ccdce45965afe1a231206a584a395fc00"}
Jul 17 22:11:40.979262 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.979212    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"
Jul 17 22:11:40.979262 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.979258    2546 scope.go:117] "RemoveContainer" containerID="aaf18f50d32c27200c2d827f2658fb2f50b632b74dc0368f45bd096462fb4e88"
Jul 17 22:11:40.996557 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.996529    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:40.998709 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:40.998690    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-jlvqg"]
Jul 17 22:11:42.536658 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:42.536627    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" path="/var/lib/kubelet/pods/2eb5cb29-a396-440d-b0eb-f929ef4c30a8/volumes"
Jul 17 22:11:44.533975 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:44.533941    2546 scope.go:117] "RemoveContainer" containerID="17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c"
Jul 17 22:11:45.002935 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:45.002900    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerStarted","Data":"1c55b1e1bece9094acd240e66af2ed9a0d2560568f3d5f87841062a719be4a18"}
Jul 17 22:11:45.021685 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:45.021640    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" podStartSLOduration=17.021628102 podStartE2EDuration="17.021628102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:45.019780564 +0000 UTC m=+3281.000151038" watchObservedRunningTime="2026-07-17 22:11:45.021628102 +0000 UTC m=+3281.001998578"
Jul 17 22:11:46.000166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000133    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:11:46.000536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000454    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000466    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000479    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000484    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000551    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000559    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.000671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.000567    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:46.003509 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.003491    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:11:46.011309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.011158    2546 generic.go:356] "Generic (PLEG): container finished" podID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerID="1c55b1e1bece9094acd240e66af2ed9a0d2560568f3d5f87841062a719be4a18" exitCode=6
Jul 17 22:11:46.011309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.011184    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerDied","Data":"1c55b1e1bece9094acd240e66af2ed9a0d2560568f3d5f87841062a719be4a18"}
Jul 17 22:11:46.011309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.011230    2546 scope.go:117] "RemoveContainer" containerID="17932efeb72c083828ab55e2111631f3fd03601a98e73ad80d1253f320ba899c"
Jul 17 22:11:46.011910 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.011888    2546 scope.go:117] "RemoveContainer" containerID="1c55b1e1bece9094acd240e66af2ed9a0d2560568f3d5f87841062a719be4a18"
Jul 17 22:11:46.012130 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:46.012105    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4_odh-ai-gateway-infra(4db6116a-b165-4911-9524-7a9dd3a64efb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb"
Jul 17 22:11:46.012776 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.012758    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:11:46.030148 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.030121    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:46.057860 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.057824    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj\" (UID: \"3a3a9800-0e7f-4c65-ac89-0110ffb281d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:11:46.158495 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.158467    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj\" (UID: \"3a3a9800-0e7f-4c65-ac89-0110ffb281d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:11:46.159052 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.159036    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj\" (UID: \"3a3a9800-0e7f-4c65-ac89-0110ffb281d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:11:46.315414 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.315346    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:11:46.656518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:46.656491    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:11:46.657973 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:11:46.657948    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a3a9800_0e7f_4c65_ac89_0110ffb281d6.slice/crio-9227c4612e6fdb0d16356d26f75216125b83ac4ef3ade2acd71e29662b5ef211 WatchSource:0}: Error finding container 9227c4612e6fdb0d16356d26f75216125b83ac4ef3ade2acd71e29662b5ef211: Status 404 returned error can't find the container with id 9227c4612e6fdb0d16356d26f75216125b83ac4ef3ade2acd71e29662b5ef211
Jul 17 22:11:47.017679 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.017649    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerStarted","Data":"72ab720f58feb2f10f052956c2151e9ff9365c88e461a88aab337fa260b2adce"}
Jul 17 22:11:47.018043 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.017712    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerStarted","Data":"9227c4612e6fdb0d16356d26f75216125b83ac4ef3ade2acd71e29662b5ef211"}
Jul 17 22:11:47.035692 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.035647    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" podStartSLOduration=2.035630915 podStartE2EDuration="2.035630915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:47.034376171 +0000 UTC m=+3283.014746649" watchObservedRunningTime="2026-07-17 22:11:47.035630915 +0000 UTC m=+3283.016001394"
Jul 17 22:11:47.145116 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.145093    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:47.166750 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.166707    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca\") pod \"4db6116a-b165-4911-9524-7a9dd3a64efb\" (UID: \"4db6116a-b165-4911-9524-7a9dd3a64efb\") "
Jul 17 22:11:47.167006 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.166984    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4db6116a-b165-4911-9524-7a9dd3a64efb" (UID: "4db6116a-b165-4911-9524-7a9dd3a64efb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:11:47.267192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:47.267153    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4db6116a-b165-4911-9524-7a9dd3a64efb-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:11:48.026279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.026250    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"
Jul 17 22:11:48.026743 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.026288    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4" event={"ID":"4db6116a-b165-4911-9524-7a9dd3a64efb","Type":"ContainerDied","Data":"906410ab57b9cbe83dfa15c796c479b917d6719d800ec59ed6970f85d76a07f7"}
Jul 17 22:11:48.026743 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.026360    2546 scope.go:117] "RemoveContainer" containerID="1c55b1e1bece9094acd240e66af2ed9a0d2560568f3d5f87841062a719be4a18"
Jul 17 22:11:48.028040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.028012    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerID="72ab720f58feb2f10f052956c2151e9ff9365c88e461a88aab337fa260b2adce" exitCode=6
Jul 17 22:11:48.028166 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.028045    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerDied","Data":"72ab720f58feb2f10f052956c2151e9ff9365c88e461a88aab337fa260b2adce"}
Jul 17 22:11:48.028269 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.028251    2546 scope.go:117] "RemoveContainer" containerID="72ab720f58feb2f10f052956c2151e9ff9365c88e461a88aab337fa260b2adce"
Jul 17 22:11:48.059144 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.059119    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:48.064523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.064501    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-ldqf4"]
Jul 17 22:11:48.536336 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:48.536267    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" path="/var/lib/kubelet/pods/4db6116a-b165-4911-9524-7a9dd3a64efb/volumes"
Jul 17 22:11:49.036513 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:49.036486    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerID="3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba" exitCode=6
Jul 17 22:11:49.036960 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:49.036533    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerDied","Data":"3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba"}
Jul 17 22:11:49.036960 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:49.036579    2546 scope.go:117] "RemoveContainer" containerID="72ab720f58feb2f10f052956c2151e9ff9365c88e461a88aab337fa260b2adce"
Jul 17 22:11:49.036960 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:49.036760    2546 scope.go:117] "RemoveContainer" containerID="3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba"
Jul 17 22:11:49.036960 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:49.036942    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj_odh-ai-gateway-infra(3a3a9800-0e7f-4c65-ac89-0110ffb281d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6"
Jul 17 22:11:50.043129 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:50.043100    2546 scope.go:117] "RemoveContainer" containerID="3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba"
Jul 17 22:11:50.043506 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:50.043253    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj_odh-ai-gateway-infra(3a3a9800-0e7f-4c65-ac89-0110ffb281d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6"
Jul 17 22:11:51.197754 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.197674    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:11:51.198224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198176    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198224 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198195    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198226    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198235    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198250    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:51.198334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198260    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="2eb5cb29-a396-440d-b0eb-f929ef4c30a8" containerName="revoke-keys"
Jul 17 22:11:51.198558 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198350    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198558 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198362    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.198558 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.198372    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:11:51.202804 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.202777    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:11:51.209573 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.209548    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:11:51.292997 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.292964    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79\" (UID: \"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:11:51.393629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.393596    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79\" (UID: \"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:11:51.394188 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.394172    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79\" (UID: \"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:11:51.515233 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.515199    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:11:51.648487 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:51.648453    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:11:51.654046 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:11:51.654022    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3cd0e2_7d8e_4e61_9cbe_29fc52ee625a.slice/crio-963628d29897c2f3e443286e149f1a3071495c8655820e5c81af41fdcddae59d WatchSource:0}: Error finding container 963628d29897c2f3e443286e149f1a3071495c8655820e5c81af41fdcddae59d: Status 404 returned error can't find the container with id 963628d29897c2f3e443286e149f1a3071495c8655820e5c81af41fdcddae59d
Jul 17 22:11:52.054410 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:52.054382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerStarted","Data":"b9af29ae3d4f8aa622945a7279ceeeaa27f2273eb62c9f6a5db2c4d0c0fff180"}
Jul 17 22:11:52.054526 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:52.054442    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerStarted","Data":"963628d29897c2f3e443286e149f1a3071495c8655820e5c81af41fdcddae59d"}
Jul 17 22:11:52.071758 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:52.071696    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" podStartSLOduration=1.071679765 podStartE2EDuration="1.071679765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:11:52.070940375 +0000 UTC m=+3288.051310852" watchObservedRunningTime="2026-07-17 22:11:52.071679765 +0000 UTC m=+3288.052050241"
Jul 17 22:11:53.060921 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:53.060886    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerID="b9af29ae3d4f8aa622945a7279ceeeaa27f2273eb62c9f6a5db2c4d0c0fff180" exitCode=6
Jul 17 22:11:53.061439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:53.060963    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerDied","Data":"b9af29ae3d4f8aa622945a7279ceeeaa27f2273eb62c9f6a5db2c4d0c0fff180"}
Jul 17 22:11:53.061439 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:53.061091    2546 scope.go:117] "RemoveContainer" containerID="b9af29ae3d4f8aa622945a7279ceeeaa27f2273eb62c9f6a5db2c4d0c0fff180"
Jul 17 22:11:54.068398 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:54.068330    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerID="93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c" exitCode=6
Jul 17 22:11:54.068774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:54.068423    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerDied","Data":"93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c"}
Jul 17 22:11:54.068774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:54.068478    2546 scope.go:117] "RemoveContainer" containerID="b9af29ae3d4f8aa622945a7279ceeeaa27f2273eb62c9f6a5db2c4d0c0fff180"
Jul 17 22:11:54.068774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:54.068654    2546 scope.go:117] "RemoveContainer" containerID="93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c"
Jul 17 22:11:54.068968 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:54.068890    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79_odh-ai-gateway-infra(bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a"
Jul 17 22:11:55.076617 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:11:55.076587    2546 scope.go:117] "RemoveContainer" containerID="93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c"
Jul 17 22:11:55.077040 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:11:55.076797    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79_odh-ai-gateway-infra(bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a"
Jul 17 22:12:02.531060 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:02.531031    2546 scope.go:117] "RemoveContainer" containerID="3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba"
Jul 17 22:12:03.121147 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:03.121068    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerStarted","Data":"9c3cb7057a84eb44f048bc4b7c9b68ad176ba44a9b169210a74bc84082ebe4ce"}
Jul 17 22:12:04.129951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.129912    2546 generic.go:356] "Generic (PLEG): container finished" podID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerID="9c3cb7057a84eb44f048bc4b7c9b68ad176ba44a9b169210a74bc84082ebe4ce" exitCode=6
Jul 17 22:12:04.130414 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.129995    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerDied","Data":"9c3cb7057a84eb44f048bc4b7c9b68ad176ba44a9b169210a74bc84082ebe4ce"}
Jul 17 22:12:04.130414 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.130076    2546 scope.go:117] "RemoveContainer" containerID="3b13d8b96e2a77c264833b51fa95eadc2bf913ffac9927584bb715e359a007ba"
Jul 17 22:12:04.130414 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.130379    2546 scope.go:117] "RemoveContainer" containerID="9c3cb7057a84eb44f048bc4b7c9b68ad176ba44a9b169210a74bc84082ebe4ce"
Jul 17 22:12:04.130604 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:04.130583    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj_odh-ai-gateway-infra(3a3a9800-0e7f-4c65-ac89-0110ffb281d6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6"
Jul 17 22:12:04.146895 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.146867    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:12:04.755796 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.755769    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:12:04.765303 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:04.765281    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-49bsn_d347b8ae-30f7-41b6-b5a9-b3adbb9f2299/ovn-acl-logging/0.log"
Jul 17 22:12:05.266295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:05.266274    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:12:05.302761 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:05.302735    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca\") pod \"3a3a9800-0e7f-4c65-ac89-0110ffb281d6\" (UID: \"3a3a9800-0e7f-4c65-ac89-0110ffb281d6\") "
Jul 17 22:12:05.303055 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:05.303035    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a3a9800-0e7f-4c65-ac89-0110ffb281d6" (UID: "3a3a9800-0e7f-4c65-ac89-0110ffb281d6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:05.404309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:05.404224    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a3a9800-0e7f-4c65-ac89-0110ffb281d6-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:12:06.143374 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.143334    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj" event={"ID":"3a3a9800-0e7f-4c65-ac89-0110ffb281d6","Type":"ContainerDied","Data":"9227c4612e6fdb0d16356d26f75216125b83ac4ef3ade2acd71e29662b5ef211"}
Jul 17 22:12:06.143552 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.143406    2546 scope.go:117] "RemoveContainer" containerID="9c3cb7057a84eb44f048bc4b7c9b68ad176ba44a9b169210a74bc84082ebe4ce"
Jul 17 22:12:06.143552 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.143353    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"
Jul 17 22:12:06.168686 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.168662    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:12:06.170774 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.170754    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-jxgjj"]
Jul 17 22:12:06.536477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:06.536444    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" path="/var/lib/kubelet/pods/3a3a9800-0e7f-4c65-ac89-0110ffb281d6/volumes"
Jul 17 22:12:10.531809 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:10.531775    2546 scope.go:117] "RemoveContainer" containerID="93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c"
Jul 17 22:12:11.174223 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:11.174197    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerStarted","Data":"d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652"}
Jul 17 22:12:11.174512 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:11.174495    2546 scope.go:117] "RemoveContainer" containerID="d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652"
Jul 17 22:12:11.174680 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:11.174659    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79_odh-ai-gateway-infra(bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a"
Jul 17 22:12:12.182567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:12.182528    2546 generic.go:356] "Generic (PLEG): container finished" podID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerID="d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652" exitCode=6
Jul 17 22:12:12.182999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:12.182600    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerDied","Data":"d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652"}
Jul 17 22:12:12.182999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:12.182670    2546 scope.go:117] "RemoveContainer" containerID="93d1e68f433b96d57eefd316fcac2a10e010dbdfa3a9961fafdf771175cf270c"
Jul 17 22:12:12.182999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:12.182966    2546 scope.go:117] "RemoveContainer" containerID="d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652"
Jul 17 22:12:12.183226 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:12.183205    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79_odh-ai-gateway-infra(bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a"
Jul 17 22:12:12.199430 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:12.199406    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:12:13.320327 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:13.320292    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:12:13.363265 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:13.363239    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca\") pod \"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a\" (UID: \"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a\") "
Jul 17 22:12:13.363551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:13.363532    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" (UID: "bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:13.463939 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:13.463878    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:12:14.195755 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.195709    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"
Jul 17 22:12:14.195931 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.195757    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79" event={"ID":"bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a","Type":"ContainerDied","Data":"963628d29897c2f3e443286e149f1a3071495c8655820e5c81af41fdcddae59d"}
Jul 17 22:12:14.195931 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.195823    2546 scope.go:117] "RemoveContainer" containerID="d4fa00aa2da5ea889b9c6cf3e01af1fe3fff3c153e8bc7637ed2f6934f18c652"
Jul 17 22:12:14.218880 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.218854    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:12:14.227230 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.227209    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-s8l79"]
Jul 17 22:12:14.536447 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:14.536414    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" path="/var/lib/kubelet/pods/bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a/volumes"
Jul 17 22:12:18.566807 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.566770    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567187    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567205    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567223    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567230    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567240    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567247    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567261    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567268    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567287 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567289    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567297    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="4db6116a-b165-4911-9524-7a9dd3a64efb" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567311    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567319    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567400    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567413    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567423    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567431    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.567815 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.567439    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:18.570565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.570544    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:18.572933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.572917    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:12:18.577269 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.577248    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:18.600930 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.600899    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd\" (UID: \"23d922d4-002f-410e-b924-8e26764fcf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:18.702253 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.702221    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd\" (UID: \"23d922d4-002f-410e-b924-8e26764fcf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:18.702834 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.702812    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd\" (UID: \"23d922d4-002f-410e-b924-8e26764fcf19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:18.882634 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:18.882572    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:19.224684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.224652    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:19.226143 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:12:19.226117    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23d922d4_002f_410e_b924_8e26764fcf19.slice/crio-8a0158c608d4226ce4aac971645d8c5a52675622a8438c974b12b10f7cb8e3b4 WatchSource:0}: Error finding container 8a0158c608d4226ce4aac971645d8c5a52675622a8438c974b12b10f7cb8e3b4: Status 404 returned error can't find the container with id 8a0158c608d4226ce4aac971645d8c5a52675622a8438c974b12b10f7cb8e3b4
Jul 17 22:12:19.428853 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.428822    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:19.429135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.429123    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:19.429186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.429137    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3cd0e2-7d8e-4e61-9cbe-29fc52ee625a" containerName="revoke-keys"
Jul 17 22:12:19.429235 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.429198    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a3a9800-0e7f-4c65-ac89-0110ffb281d6" containerName="revoke-keys"
Jul 17 22:12:19.433150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.433134    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:19.439532 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.439489    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:19.508795 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.508759    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86\" (UID: \"672a9452-afe0-446f-acf2-76981488c779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:19.609228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.609195    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86\" (UID: \"672a9452-afe0-446f-acf2-76981488c779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:19.609868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.609847    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86\" (UID: \"672a9452-afe0-446f-acf2-76981488c779\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:19.744244 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.744214    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:19.876711 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:19.876688    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:19.912613 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:12:19.912580    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672a9452_afe0_446f_acf2_76981488c779.slice/crio-1a171c60873a608686f6b58eb489437fb9913cb008f2c6e38e0e920c770495d8 WatchSource:0}: Error finding container 1a171c60873a608686f6b58eb489437fb9913cb008f2c6e38e0e920c770495d8: Status 404 returned error can't find the container with id 1a171c60873a608686f6b58eb489437fb9913cb008f2c6e38e0e920c770495d8
Jul 17 22:12:20.231694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.231658    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerStarted","Data":"a77d42539ac80646e34dba492dd40312e33bc79d755e1b485f9eb49f027af895"}
Jul 17 22:12:20.231841 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.231734    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerStarted","Data":"1a171c60873a608686f6b58eb489437fb9913cb008f2c6e38e0e920c770495d8"}
Jul 17 22:12:20.233114 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.233092    2546 generic.go:356] "Generic (PLEG): container finished" podID="23d922d4-002f-410e-b924-8e26764fcf19" containerID="e44aaf3796e2c91050a251d6a745959ce66e8c5ed27f4f982bf38091b01c216d" exitCode=6
Jul 17 22:12:20.233206 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.233176    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" event={"ID":"23d922d4-002f-410e-b924-8e26764fcf19","Type":"ContainerDied","Data":"e44aaf3796e2c91050a251d6a745959ce66e8c5ed27f4f982bf38091b01c216d"}
Jul 17 22:12:20.233285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.233217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" event={"ID":"23d922d4-002f-410e-b924-8e26764fcf19","Type":"ContainerStarted","Data":"8a0158c608d4226ce4aac971645d8c5a52675622a8438c974b12b10f7cb8e3b4"}
Jul 17 22:12:20.233333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.233321    2546 scope.go:117] "RemoveContainer" containerID="e44aaf3796e2c91050a251d6a745959ce66e8c5ed27f4f982bf38091b01c216d"
Jul 17 22:12:20.248303 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:20.248256    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" podStartSLOduration=1.24823962 podStartE2EDuration="1.24823962s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:12:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:12:20.247356627 +0000 UTC m=+3316.227727106" watchObservedRunningTime="2026-07-17 22:12:20.24823962 +0000 UTC m=+3316.228610099"
Jul 17 22:12:21.240041 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.240007    2546 generic.go:356] "Generic (PLEG): container finished" podID="672a9452-afe0-446f-acf2-76981488c779" containerID="a77d42539ac80646e34dba492dd40312e33bc79d755e1b485f9eb49f027af895" exitCode=6
Jul 17 22:12:21.240477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.240096    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerDied","Data":"a77d42539ac80646e34dba492dd40312e33bc79d755e1b485f9eb49f027af895"}
Jul 17 22:12:21.240477 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.240299    2546 scope.go:117] "RemoveContainer" containerID="a77d42539ac80646e34dba492dd40312e33bc79d755e1b485f9eb49f027af895"
Jul 17 22:12:21.241959 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.241939    2546 generic.go:356] "Generic (PLEG): container finished" podID="23d922d4-002f-410e-b924-8e26764fcf19" containerID="52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba" exitCode=6
Jul 17 22:12:21.242062 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.242029    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" event={"ID":"23d922d4-002f-410e-b924-8e26764fcf19","Type":"ContainerDied","Data":"52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba"}
Jul 17 22:12:21.242115 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.242081    2546 scope.go:117] "RemoveContainer" containerID="e44aaf3796e2c91050a251d6a745959ce66e8c5ed27f4f982bf38091b01c216d"
Jul 17 22:12:21.242187 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:21.242167    2546 scope.go:117] "RemoveContainer" containerID="52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba"
Jul 17 22:12:21.242365 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:21.242343    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd_odh-ai-gateway-infra(23d922d4-002f-410e-b924-8e26764fcf19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" podUID="23d922d4-002f-410e-b924-8e26764fcf19"
Jul 17 22:12:22.249389 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:22.249352    2546 generic.go:356] "Generic (PLEG): container finished" podID="672a9452-afe0-446f-acf2-76981488c779" containerID="b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41" exitCode=6
Jul 17 22:12:22.249866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:22.249441    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerDied","Data":"b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41"}
Jul 17 22:12:22.249866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:22.249516    2546 scope.go:117] "RemoveContainer" containerID="a77d42539ac80646e34dba492dd40312e33bc79d755e1b485f9eb49f027af895"
Jul 17 22:12:22.249866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:22.249781    2546 scope.go:117] "RemoveContainer" containerID="b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41"
Jul 17 22:12:22.250042 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:22.249957    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86_odh-ai-gateway-infra(672a9452-afe0-446f-acf2-76981488c779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" podUID="672a9452-afe0-446f-acf2-76981488c779"
Jul 17 22:12:22.251205 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:22.251185    2546 scope.go:117] "RemoveContainer" containerID="52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba"
Jul 17 22:12:22.251332 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:22.251317    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd_odh-ai-gateway-infra(23d922d4-002f-410e-b924-8e26764fcf19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" podUID="23d922d4-002f-410e-b924-8e26764fcf19"
Jul 17 22:12:23.263675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:23.263648    2546 scope.go:117] "RemoveContainer" containerID="b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41"
Jul 17 22:12:23.264080 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:23.263932    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86_odh-ai-gateway-infra(672a9452-afe0-446f-acf2-76981488c779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" podUID="672a9452-afe0-446f-acf2-76981488c779"
Jul 17 22:12:33.531643 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:33.531607    2546 scope.go:117] "RemoveContainer" containerID="b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41"
Jul 17 22:12:34.323315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:34.323277    2546 generic.go:356] "Generic (PLEG): container finished" podID="672a9452-afe0-446f-acf2-76981488c779" containerID="ff93521b2cf260028bb0953702abff15c31a11e8c9ff62c5eaa0e4c5dce247c3" exitCode=6
Jul 17 22:12:34.323488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:34.323348    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerDied","Data":"ff93521b2cf260028bb0953702abff15c31a11e8c9ff62c5eaa0e4c5dce247c3"}
Jul 17 22:12:34.323488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:34.323421    2546 scope.go:117] "RemoveContainer" containerID="b078896dc7ffd5125260f2e22b19daa389179bd21ca2b065aab764833aa75e41"
Jul 17 22:12:34.323778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:34.323758    2546 scope.go:117] "RemoveContainer" containerID="ff93521b2cf260028bb0953702abff15c31a11e8c9ff62c5eaa0e4c5dce247c3"
Jul 17 22:12:34.324068 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:34.323982    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86_odh-ai-gateway-infra(672a9452-afe0-446f-acf2-76981488c779)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" podUID="672a9452-afe0-446f-acf2-76981488c779"
Jul 17 22:12:34.533273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:34.533246    2546 scope.go:117] "RemoveContainer" containerID="52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba"
Jul 17 22:12:35.331708 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.331672    2546 generic.go:356] "Generic (PLEG): container finished" podID="23d922d4-002f-410e-b924-8e26764fcf19" containerID="2fa5f1e73eb76dd4c3c85b79f0482470e020ddbcd14d9d99944ab8b97b74cf99" exitCode=6
Jul 17 22:12:35.331914 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.331752    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" event={"ID":"23d922d4-002f-410e-b924-8e26764fcf19","Type":"ContainerDied","Data":"2fa5f1e73eb76dd4c3c85b79f0482470e020ddbcd14d9d99944ab8b97b74cf99"}
Jul 17 22:12:35.331914 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.331808    2546 scope.go:117] "RemoveContainer" containerID="52228f077702b7c99f1eacd213cc5e37c697aec040be4c88430be71d174f39ba"
Jul 17 22:12:35.332119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.332104    2546 scope.go:117] "RemoveContainer" containerID="2fa5f1e73eb76dd4c3c85b79f0482470e020ddbcd14d9d99944ab8b97b74cf99"
Jul 17 22:12:35.332357 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:35.332337    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd_odh-ai-gateway-infra(23d922d4-002f-410e-b924-8e26764fcf19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" podUID="23d922d4-002f-410e-b924-8e26764fcf19"
Jul 17 22:12:35.353202 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.353176    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:35.487485 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.487464    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:35.641405 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.641378    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca\") pod \"672a9452-afe0-446f-acf2-76981488c779\" (UID: \"672a9452-afe0-446f-acf2-76981488c779\") "
Jul 17 22:12:35.641808 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.641694    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "672a9452-afe0-446f-acf2-76981488c779" (UID: "672a9452-afe0-446f-acf2-76981488c779"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:35.742209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:35.742178    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/672a9452-afe0-446f-acf2-76981488c779-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:12:36.343833 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.343803    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"
Jul 17 22:12:36.344039 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.343802    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86" event={"ID":"672a9452-afe0-446f-acf2-76981488c779","Type":"ContainerDied","Data":"1a171c60873a608686f6b58eb489437fb9913cb008f2c6e38e0e920c770495d8"}
Jul 17 22:12:36.344039 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.343951    2546 scope.go:117] "RemoveContainer" containerID="ff93521b2cf260028bb0953702abff15c31a11e8c9ff62c5eaa0e4c5dce247c3"
Jul 17 22:12:36.356291 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.356267    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:36.368399 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.368372    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:36.377651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.377625    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-b7q86"]
Jul 17 22:12:36.487835 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.487813    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:36.535576 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.535550    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="672a9452-afe0-446f-acf2-76981488c779" path="/var/lib/kubelet/pods/672a9452-afe0-446f-acf2-76981488c779/volumes"
Jul 17 22:12:36.548459 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.548437    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca\") pod \"23d922d4-002f-410e-b924-8e26764fcf19\" (UID: \"23d922d4-002f-410e-b924-8e26764fcf19\") "
Jul 17 22:12:36.548850 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.548825    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23d922d4-002f-410e-b924-8e26764fcf19" (UID: "23d922d4-002f-410e-b924-8e26764fcf19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:12:36.649587 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:36.649529    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23d922d4-002f-410e-b924-8e26764fcf19-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:12:37.353813 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:37.353671    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"
Jul 17 22:12:37.353813 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:37.353689    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd" event={"ID":"23d922d4-002f-410e-b924-8e26764fcf19","Type":"ContainerDied","Data":"8a0158c608d4226ce4aac971645d8c5a52675622a8438c974b12b10f7cb8e3b4"}
Jul 17 22:12:37.353813 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:37.353773    2546 scope.go:117] "RemoveContainer" containerID="2fa5f1e73eb76dd4c3c85b79f0482470e020ddbcd14d9d99944ab8b97b74cf99"
Jul 17 22:12:37.376412 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:37.376293    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:37.382444 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:37.382422    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-2cmmd"]
Jul 17 22:12:38.536079 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:38.536049    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23d922d4-002f-410e-b924-8e26764fcf19" path="/var/lib/kubelet/pods/23d922d4-002f-410e-b924-8e26764fcf19/volumes"
Jul 17 22:12:46.041190 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041158    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041444    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041454    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041462    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041468    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041476    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041482    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041494    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041499    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041506    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041511    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.041556 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041559    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.042024 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041567    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.042024 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041572    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.042024 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041579    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:46.042024 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.041586    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:46.044525 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.044506    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:12:46.046777 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.046760    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:12:46.052541 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.052519    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:12:46.120258 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.120227    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4\" (UID: \"00d29959-6a98-446a-92b5-47cb204ed197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:12:46.220905 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.220875    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4\" (UID: \"00d29959-6a98-446a-92b5-47cb204ed197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:12:46.221431 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.221414    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4\" (UID: \"00d29959-6a98-446a-92b5-47cb204ed197\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:12:46.355497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.355417    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:12:46.485693 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:46.485656    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:12:46.488593 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:12:46.488566    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00d29959_6a98_446a_92b5_47cb204ed197.slice/crio-7be13fea59f8a07b497f765b80132977ac1ef44c0acf0ed4120add3628f0735c WatchSource:0}: Error finding container 7be13fea59f8a07b497f765b80132977ac1ef44c0acf0ed4120add3628f0735c: Status 404 returned error can't find the container with id 7be13fea59f8a07b497f765b80132977ac1ef44c0acf0ed4120add3628f0735c
Jul 17 22:12:47.407898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:47.407824    2546 generic.go:356] "Generic (PLEG): container finished" podID="00d29959-6a98-446a-92b5-47cb204ed197" containerID="d34734b40da8e1421aeeb8296beb904eb05e67a32f6577ce4aee18ff80893909" exitCode=6
Jul 17 22:12:47.407898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:47.407883    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" event={"ID":"00d29959-6a98-446a-92b5-47cb204ed197","Type":"ContainerDied","Data":"d34734b40da8e1421aeeb8296beb904eb05e67a32f6577ce4aee18ff80893909"}
Jul 17 22:12:47.408275 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:47.407922    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" event={"ID":"00d29959-6a98-446a-92b5-47cb204ed197","Type":"ContainerStarted","Data":"7be13fea59f8a07b497f765b80132977ac1ef44c0acf0ed4120add3628f0735c"}
Jul 17 22:12:47.408275 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:47.408064    2546 scope.go:117] "RemoveContainer" containerID="d34734b40da8e1421aeeb8296beb904eb05e67a32f6577ce4aee18ff80893909"
Jul 17 22:12:48.414618 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:48.414584    2546 generic.go:356] "Generic (PLEG): container finished" podID="00d29959-6a98-446a-92b5-47cb204ed197" containerID="85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2" exitCode=6
Jul 17 22:12:48.415097 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:48.414650    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" event={"ID":"00d29959-6a98-446a-92b5-47cb204ed197","Type":"ContainerDied","Data":"85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2"}
Jul 17 22:12:48.415097 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:48.414772    2546 scope.go:117] "RemoveContainer" containerID="d34734b40da8e1421aeeb8296beb904eb05e67a32f6577ce4aee18ff80893909"
Jul 17 22:12:48.415097 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:48.414911    2546 scope.go:117] "RemoveContainer" containerID="85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2"
Jul 17 22:12:48.415211 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:48.415100    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4_odh-ai-gateway-infra(00d29959-6a98-446a-92b5-47cb204ed197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" podUID="00d29959-6a98-446a-92b5-47cb204ed197"
Jul 17 22:12:49.423150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:49.423116    2546 scope.go:117] "RemoveContainer" containerID="85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2"
Jul 17 22:12:49.423579 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:49.423295    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4_odh-ai-gateway-infra(00d29959-6a98-446a-92b5-47cb204ed197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" podUID="00d29959-6a98-446a-92b5-47cb204ed197"
Jul 17 22:12:51.242119 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.242080    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:12:51.242513 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.242453    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:51.242513 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.242467    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="23d922d4-002f-410e-b924-8e26764fcf19" containerName="revoke-keys"
Jul 17 22:12:51.242596 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.242537    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="672a9452-afe0-446f-acf2-76981488c779" containerName="revoke-keys"
Jul 17 22:12:51.246662 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.246642    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:12:51.252610 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.252580    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:12:51.367181 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.367157    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c\" (UID: \"85bf6a40-0511-43f5-999b-035b1c017ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:12:51.468172 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.468142    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c\" (UID: \"85bf6a40-0511-43f5-999b-035b1c017ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:12:51.468794 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.468770    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c\" (UID: \"85bf6a40-0511-43f5-999b-035b1c017ef5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:12:51.561078 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.561011    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:12:51.691667 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:51.691642    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:12:51.692317 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:12:51.692299    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85bf6a40_0511_43f5_999b_035b1c017ef5.slice/crio-09043e33883bdc04384c7da9241cabd3392cd64805c591c1b2e02d5fc0f13709 WatchSource:0}: Error finding container 09043e33883bdc04384c7da9241cabd3392cd64805c591c1b2e02d5fc0f13709: Status 404 returned error can't find the container with id 09043e33883bdc04384c7da9241cabd3392cd64805c591c1b2e02d5fc0f13709
Jul 17 22:12:52.442083 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:52.442053    2546 generic.go:356] "Generic (PLEG): container finished" podID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerID="845c840a26f101dcbecea563c0bd79da84320a12ab21b5e6bf09527a4c5800de" exitCode=6
Jul 17 22:12:52.442381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:52.442097    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" event={"ID":"85bf6a40-0511-43f5-999b-035b1c017ef5","Type":"ContainerDied","Data":"845c840a26f101dcbecea563c0bd79da84320a12ab21b5e6bf09527a4c5800de"}
Jul 17 22:12:52.442381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:52.442136    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" event={"ID":"85bf6a40-0511-43f5-999b-035b1c017ef5","Type":"ContainerStarted","Data":"09043e33883bdc04384c7da9241cabd3392cd64805c591c1b2e02d5fc0f13709"}
Jul 17 22:12:52.442381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:52.442299    2546 scope.go:117] "RemoveContainer" containerID="845c840a26f101dcbecea563c0bd79da84320a12ab21b5e6bf09527a4c5800de"
Jul 17 22:12:53.449674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:53.449574    2546 generic.go:356] "Generic (PLEG): container finished" podID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerID="96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d" exitCode=6
Jul 17 22:12:53.450150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:53.449663    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" event={"ID":"85bf6a40-0511-43f5-999b-035b1c017ef5","Type":"ContainerDied","Data":"96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d"}
Jul 17 22:12:53.450150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:53.449779    2546 scope.go:117] "RemoveContainer" containerID="845c840a26f101dcbecea563c0bd79da84320a12ab21b5e6bf09527a4c5800de"
Jul 17 22:12:53.450150 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:53.449981    2546 scope.go:117] "RemoveContainer" containerID="96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d"
Jul 17 22:12:53.450293 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:53.450192    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c_odh-ai-gateway-infra(85bf6a40-0511-43f5-999b-035b1c017ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5"
Jul 17 22:12:54.457173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:12:54.457135    2546 scope.go:117] "RemoveContainer" containerID="96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d"
Jul 17 22:12:54.457628 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:12:54.457308    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c_odh-ai-gateway-infra(85bf6a40-0511-43f5-999b-035b1c017ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5"
Jul 17 22:13:00.531019 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:00.530991    2546 scope.go:117] "RemoveContainer" containerID="85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2"
Jul 17 22:13:01.494712 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:01.494674    2546 generic.go:356] "Generic (PLEG): container finished" podID="00d29959-6a98-446a-92b5-47cb204ed197" containerID="060792fd4288b6c3285103f9d9d16c71d41b9eed7df801f9a39e2496ed16a4e1" exitCode=6
Jul 17 22:13:01.494871 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:01.494750    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" event={"ID":"00d29959-6a98-446a-92b5-47cb204ed197","Type":"ContainerDied","Data":"060792fd4288b6c3285103f9d9d16c71d41b9eed7df801f9a39e2496ed16a4e1"}
Jul 17 22:13:01.494871 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:01.494809    2546 scope.go:117] "RemoveContainer" containerID="85eb5b88eea82714c4a7d95bd5e5ae76145f3f65aa1fa949a1f39b21eaeef3f2"
Jul 17 22:13:01.495165 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:01.495148    2546 scope.go:117] "RemoveContainer" containerID="060792fd4288b6c3285103f9d9d16c71d41b9eed7df801f9a39e2496ed16a4e1"
Jul 17 22:13:01.495377 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:01.495359    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4_odh-ai-gateway-infra(00d29959-6a98-446a-92b5-47cb204ed197)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" podUID="00d29959-6a98-446a-92b5-47cb204ed197"
Jul 17 22:13:02.519847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:02.519812    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:13:02.653032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:02.653010    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:13:02.756357 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:02.756327    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca\") pod \"00d29959-6a98-446a-92b5-47cb204ed197\" (UID: \"00d29959-6a98-446a-92b5-47cb204ed197\") "
Jul 17 22:13:02.756673 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:02.756650    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00d29959-6a98-446a-92b5-47cb204ed197" (UID: "00d29959-6a98-446a-92b5-47cb204ed197"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:02.857609 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:02.857534    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00d29959-6a98-446a-92b5-47cb204ed197-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:03.508253 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:03.508224    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"
Jul 17 22:13:03.508466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:03.508272    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4" event={"ID":"00d29959-6a98-446a-92b5-47cb204ed197","Type":"ContainerDied","Data":"7be13fea59f8a07b497f765b80132977ac1ef44c0acf0ed4120add3628f0735c"}
Jul 17 22:13:03.508466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:03.508334    2546 scope.go:117] "RemoveContainer" containerID="060792fd4288b6c3285103f9d9d16c71d41b9eed7df801f9a39e2496ed16a4e1"
Jul 17 22:13:03.533701 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:03.533676    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:13:03.542058 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:03.542036    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-gbsz4"]
Jul 17 22:13:04.537278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:04.537240    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00d29959-6a98-446a-92b5-47cb204ed197" path="/var/lib/kubelet/pods/00d29959-6a98-446a-92b5-47cb204ed197/volumes"
Jul 17 22:13:05.531500 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:05.531446    2546 scope.go:117] "RemoveContainer" containerID="96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d"
Jul 17 22:13:06.532699 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:06.532667    2546 generic.go:356] "Generic (PLEG): container finished" podID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerID="3e0ad3d55820cd9987f841813f156cecae444d717fae31a115adc704fb9655b6" exitCode=6
Jul 17 22:13:06.536144 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:06.536113    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" event={"ID":"85bf6a40-0511-43f5-999b-035b1c017ef5","Type":"ContainerDied","Data":"3e0ad3d55820cd9987f841813f156cecae444d717fae31a115adc704fb9655b6"}
Jul 17 22:13:06.536276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:06.536171    2546 scope.go:117] "RemoveContainer" containerID="96e44fe5528be5f0d22c448573d37c8a901e9f3706f16f0ee3618ee250eb511d"
Jul 17 22:13:06.536442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:06.536422    2546 scope.go:117] "RemoveContainer" containerID="3e0ad3d55820cd9987f841813f156cecae444d717fae31a115adc704fb9655b6"
Jul 17 22:13:06.536664 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:06.536640    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c_odh-ai-gateway-infra(85bf6a40-0511-43f5-999b-035b1c017ef5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5"
Jul 17 22:13:07.561481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:07.561443    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:13:07.692862 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:07.692840    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:13:07.800881 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:07.800855    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca\") pod \"85bf6a40-0511-43f5-999b-035b1c017ef5\" (UID: \"85bf6a40-0511-43f5-999b-035b1c017ef5\") "
Jul 17 22:13:07.801168 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:07.801148    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "85bf6a40-0511-43f5-999b-035b1c017ef5" (UID: "85bf6a40-0511-43f5-999b-035b1c017ef5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:07.901924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:07.901866    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/85bf6a40-0511-43f5-999b-035b1c017ef5-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:08.546212 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.546183    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"
Jul 17 22:13:08.546336 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.546217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c" event={"ID":"85bf6a40-0511-43f5-999b-035b1c017ef5","Type":"ContainerDied","Data":"09043e33883bdc04384c7da9241cabd3392cd64805c591c1b2e02d5fc0f13709"}
Jul 17 22:13:08.546336 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.546268    2546 scope.go:117] "RemoveContainer" containerID="3e0ad3d55820cd9987f841813f156cecae444d717fae31a115adc704fb9655b6"
Jul 17 22:13:08.561989 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.561967    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:13:08.567228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.567206    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-dq42c"]
Jul 17 22:13:08.607988 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.607962    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:08.608234 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608223    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608236    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608244    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608250    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608258    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608263    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608269    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608273    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608287    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608291    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608351    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608357    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608363    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:08.608497 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.608371    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:08.612424 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.612410    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:08.614755 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.614715    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:13:08.617714 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.617694    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:08.708161 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.708140    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8\" (UID: \"22978207-3f58-4c33-876c-e032df6d4420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:08.809231 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.809172    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8\" (UID: \"22978207-3f58-4c33-876c-e032df6d4420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:08.809746 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.809706    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8\" (UID: \"22978207-3f58-4c33-876c-e032df6d4420\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:08.923841 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:08.923822    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:09.052157 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.052131    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:09.053918 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:09.053891    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22978207_3f58_4c33_876c_e032df6d4420.slice/crio-4f213daef16e643687268e4707894ede54d594dd471482312f805d6a88d1c804 WatchSource:0}: Error finding container 4f213daef16e643687268e4707894ede54d594dd471482312f805d6a88d1c804: Status 404 returned error can't find the container with id 4f213daef16e643687268e4707894ede54d594dd471482312f805d6a88d1c804
Jul 17 22:13:09.473760 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.473662    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:09.474088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.474070    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:09.474175 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.474090    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:09.474237 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.474181    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="00d29959-6a98-446a-92b5-47cb204ed197" containerName="revoke-keys"
Jul 17 22:13:09.474237 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.474196    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" containerName="revoke-keys"
Jul 17 22:13:09.477218 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.477197    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:09.483820 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.483787    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:09.514847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.514822    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7\" (UID: \"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:09.553109 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.553085    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerStarted","Data":"e1b88eaa7c3e6c328cbe73691fc61a231cb5bc1ecbdec2e822dfa03c882d4aa3"}
Jul 17 22:13:09.553219 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.553131    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerStarted","Data":"4f213daef16e643687268e4707894ede54d594dd471482312f805d6a88d1c804"}
Jul 17 22:13:09.570070 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.570028    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" podStartSLOduration=1.5700162789999998 podStartE2EDuration="1.570016279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:09.566317105 +0000 UTC m=+3365.546687581" watchObservedRunningTime="2026-07-17 22:13:09.570016279 +0000 UTC m=+3365.550386776"
Jul 17 22:13:09.615959 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.615931    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7\" (UID: \"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:09.616450 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.616433    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7\" (UID: \"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:09.789033 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.789009    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:09.920272 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:09.920247    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:09.921483 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:09.921456    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee3abb0_e5fe_4dbc_ab34_ac28f83c27a1.slice/crio-d39219d446a6b3350657744275496f2cc4a2da194664988474d3045d15606d76 WatchSource:0}: Error finding container d39219d446a6b3350657744275496f2cc4a2da194664988474d3045d15606d76: Status 404 returned error can't find the container with id d39219d446a6b3350657744275496f2cc4a2da194664988474d3045d15606d76
Jul 17 22:13:10.536202 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.536160    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85bf6a40-0511-43f5-999b-035b1c017ef5" path="/var/lib/kubelet/pods/85bf6a40-0511-43f5-999b-035b1c017ef5/volumes"
Jul 17 22:13:10.560157 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.560128    2546 generic.go:356] "Generic (PLEG): container finished" podID="22978207-3f58-4c33-876c-e032df6d4420" containerID="e1b88eaa7c3e6c328cbe73691fc61a231cb5bc1ecbdec2e822dfa03c882d4aa3" exitCode=6
Jul 17 22:13:10.560293 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.560217    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerDied","Data":"e1b88eaa7c3e6c328cbe73691fc61a231cb5bc1ecbdec2e822dfa03c882d4aa3"}
Jul 17 22:13:10.560396 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.560381    2546 scope.go:117] "RemoveContainer" containerID="e1b88eaa7c3e6c328cbe73691fc61a231cb5bc1ecbdec2e822dfa03c882d4aa3"
Jul 17 22:13:10.562039 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.562014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerStarted","Data":"8728a7a02a707b47352f439e7140e93cf1527f91d7412c18e2c929a7b6bba023"}
Jul 17 22:13:10.562167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.562062    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerStarted","Data":"d39219d446a6b3350657744275496f2cc4a2da194664988474d3045d15606d76"}
Jul 17 22:13:10.590836 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:10.590794    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" podStartSLOduration=1.590782102 podStartE2EDuration="1.590782102s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:10.58892868 +0000 UTC m=+3366.569299180" watchObservedRunningTime="2026-07-17 22:13:10.590782102 +0000 UTC m=+3366.571152578"
Jul 17 22:13:11.568521 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.568489    2546 generic.go:356] "Generic (PLEG): container finished" podID="22978207-3f58-4c33-876c-e032df6d4420" containerID="0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7" exitCode=6
Jul 17 22:13:11.568716 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.568569    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerDied","Data":"0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7"}
Jul 17 22:13:11.568716 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.568650    2546 scope.go:117] "RemoveContainer" containerID="e1b88eaa7c3e6c328cbe73691fc61a231cb5bc1ecbdec2e822dfa03c882d4aa3"
Jul 17 22:13:11.568873 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.568778    2546 scope.go:117] "RemoveContainer" containerID="0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7"
Jul 17 22:13:11.568998 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:11.568977    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8_odh-ai-gateway-infra(22978207-3f58-4c33-876c-e032df6d4420)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" podUID="22978207-3f58-4c33-876c-e032df6d4420"
Jul 17 22:13:11.570249 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.570231    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerID="8728a7a02a707b47352f439e7140e93cf1527f91d7412c18e2c929a7b6bba023" exitCode=6
Jul 17 22:13:11.570381 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.570275    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerDied","Data":"8728a7a02a707b47352f439e7140e93cf1527f91d7412c18e2c929a7b6bba023"}
Jul 17 22:13:11.570466 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:11.570453    2546 scope.go:117] "RemoveContainer" containerID="8728a7a02a707b47352f439e7140e93cf1527f91d7412c18e2c929a7b6bba023"
Jul 17 22:13:12.577238 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:12.577207    2546 scope.go:117] "RemoveContainer" containerID="0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7"
Jul 17 22:13:12.577658 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:12.577421    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8_odh-ai-gateway-infra(22978207-3f58-4c33-876c-e032df6d4420)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" podUID="22978207-3f58-4c33-876c-e032df6d4420"
Jul 17 22:13:12.578926 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:12.578906    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerID="3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562" exitCode=6
Jul 17 22:13:12.579017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:12.578936    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerDied","Data":"3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562"}
Jul 17 22:13:12.579017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:12.578976    2546 scope.go:117] "RemoveContainer" containerID="8728a7a02a707b47352f439e7140e93cf1527f91d7412c18e2c929a7b6bba023"
Jul 17 22:13:12.579141 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:12.579125    2546 scope.go:117] "RemoveContainer" containerID="3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562"
Jul 17 22:13:12.579331 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:12.579311    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7_odh-ai-gateway-infra(8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1"
Jul 17 22:13:13.586067 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:13.586039    2546 scope.go:117] "RemoveContainer" containerID="3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562"
Jul 17 22:13:13.586452 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:13.586227    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7_odh-ai-gateway-infra(8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1"
Jul 17 22:13:24.534370 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:24.534336    2546 scope.go:117] "RemoveContainer" containerID="3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562"
Jul 17 22:13:25.648271 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:25.648192    2546 generic.go:356] "Generic (PLEG): container finished" podID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerID="9e84977a195cdc4a3469c205eb394745332379d40793c80acac63ed4b9e3e45b" exitCode=6
Jul 17 22:13:25.648669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:25.648267    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerDied","Data":"9e84977a195cdc4a3469c205eb394745332379d40793c80acac63ed4b9e3e45b"}
Jul 17 22:13:25.648669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:25.648328    2546 scope.go:117] "RemoveContainer" containerID="3e5664175dbf634f044916d40335df0545b67b8c3839c4032826a95a44c38562"
Jul 17 22:13:25.648669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:25.648618    2546 scope.go:117] "RemoveContainer" containerID="9e84977a195cdc4a3469c205eb394745332379d40793c80acac63ed4b9e3e45b"
Jul 17 22:13:25.648883 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:25.648831    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7_odh-ai-gateway-infra(8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1"
Jul 17 22:13:26.678405 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:26.678375    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:26.797587 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:26.797566    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:26.848248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:26.848223    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca\") pod \"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1\" (UID: \"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1\") "
Jul 17 22:13:26.848524 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:26.848504    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" (UID: "8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:26.949799 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:26.949741    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:27.531526 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.531490    2546 scope.go:117] "RemoveContainer" containerID="0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7"
Jul 17 22:13:27.660674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.660646    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7" event={"ID":"8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1","Type":"ContainerDied","Data":"d39219d446a6b3350657744275496f2cc4a2da194664988474d3045d15606d76"}
Jul 17 22:13:27.660674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.660677    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"
Jul 17 22:13:27.660907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.660704    2546 scope.go:117] "RemoveContainer" containerID="9e84977a195cdc4a3469c205eb394745332379d40793c80acac63ed4b9e3e45b"
Jul 17 22:13:27.683109 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.683081    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:27.688914 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:27.688891    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-7qxx7"]
Jul 17 22:13:28.536014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:28.535957    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" path="/var/lib/kubelet/pods/8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1/volumes"
Jul 17 22:13:28.676569 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:28.676537    2546 generic.go:356] "Generic (PLEG): container finished" podID="22978207-3f58-4c33-876c-e032df6d4420" containerID="d6fcab40c733609a0c2c55b002e1748bf0f1aa1da2e781bec45107784a032c68" exitCode=6
Jul 17 22:13:28.676739 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:28.676609    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerDied","Data":"d6fcab40c733609a0c2c55b002e1748bf0f1aa1da2e781bec45107784a032c68"}
Jul 17 22:13:28.676739 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:28.676696    2546 scope.go:117] "RemoveContainer" containerID="0f1fa400a7aaa99d9dbfa03a0c3499577d3d2cddd5e26f37aed9ec492a6585d7"
Jul 17 22:13:28.677008 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:28.676979    2546 scope.go:117] "RemoveContainer" containerID="d6fcab40c733609a0c2c55b002e1748bf0f1aa1da2e781bec45107784a032c68"
Jul 17 22:13:28.677211 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:28.677189    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8_odh-ai-gateway-infra(22978207-3f58-4c33-876c-e032df6d4420)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" podUID="22978207-3f58-4c33-876c-e032df6d4420"
Jul 17 22:13:29.276626 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.276594    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"]
Jul 17 22:13:29.277017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.276939    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.277017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.276952    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.277017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.276964    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.277017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.276969    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.277017 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.277017    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.277201 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.277027    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:29.281189 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.281171    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:29.287364 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.287343    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"]
Jul 17 22:13:29.367322 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.367294    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t\" (UID: \"0dd740e3-fd78-495f-99d9-98df1a3d3dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:29.468209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.468179    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t\" (UID: \"0dd740e3-fd78-495f-99d9-98df1a3d3dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:29.468762 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.468743    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t\" (UID: \"0dd740e3-fd78-495f-99d9-98df1a3d3dca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:29.593071 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.593017    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:29.702514 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.702480    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:29.724389 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.724364    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"]
Jul 17 22:13:29.725651 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:29.725624    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dd740e3_fd78_495f_99d9_98df1a3d3dca.slice/crio-07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51 WatchSource:0}: Error finding container 07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51: Status 404 returned error can't find the container with id 07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51
Jul 17 22:13:29.845426 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.845373    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:29.982712 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.982683    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca\") pod \"22978207-3f58-4c33-876c-e032df6d4420\" (UID: \"22978207-3f58-4c33-876c-e032df6d4420\") "
Jul 17 22:13:29.983053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:29.983029    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "22978207-3f58-4c33-876c-e032df6d4420" (UID: "22978207-3f58-4c33-876c-e032df6d4420"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:30.083650 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.083624    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/22978207-3f58-4c33-876c-e032df6d4420-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:30.692896 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.692825    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"
Jul 17 22:13:30.692896 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.692833    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8" event={"ID":"22978207-3f58-4c33-876c-e032df6d4420","Type":"ContainerDied","Data":"4f213daef16e643687268e4707894ede54d594dd471482312f805d6a88d1c804"}
Jul 17 22:13:30.693318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.692903    2546 scope.go:117] "RemoveContainer" containerID="d6fcab40c733609a0c2c55b002e1748bf0f1aa1da2e781bec45107784a032c68"
Jul 17 22:13:30.694771 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.694744    2546 generic.go:356] "Generic (PLEG): container finished" podID="0dd740e3-fd78-495f-99d9-98df1a3d3dca" containerID="21b9e51623ceaf7a33461cc079056a0eb21223270843ea6b63befc889d9bf106" exitCode=0
Jul 17 22:13:30.694894 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.694837    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t" event={"ID":"0dd740e3-fd78-495f-99d9-98df1a3d3dca","Type":"ContainerDied","Data":"21b9e51623ceaf7a33461cc079056a0eb21223270843ea6b63befc889d9bf106"}
Jul 17 22:13:30.694894 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.694880    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t" event={"ID":"0dd740e3-fd78-495f-99d9-98df1a3d3dca","Type":"ContainerStarted","Data":"07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51"}
Jul 17 22:13:30.710120 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.710099    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:30.716291 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:30.716269    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-xptj8"]
Jul 17 22:13:31.832315 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:31.832293    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:31.898342 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:31.898318    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca\") pod \"0dd740e3-fd78-495f-99d9-98df1a3d3dca\" (UID: \"0dd740e3-fd78-495f-99d9-98df1a3d3dca\") "
Jul 17 22:13:31.898645 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:31.898622    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0dd740e3-fd78-495f-99d9-98df1a3d3dca" (UID: "0dd740e3-fd78-495f-99d9-98df1a3d3dca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:31.999347 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:31.999319    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0dd740e3-fd78-495f-99d9-98df1a3d3dca-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:32.537029 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:32.536997    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22978207-3f58-4c33-876c-e032df6d4420" path="/var/lib/kubelet/pods/22978207-3f58-4c33-876c-e032df6d4420/volumes"
Jul 17 22:13:32.709354 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:32.709328    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t"
Jul 17 22:13:32.709354 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:32.709341    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t" event={"ID":"0dd740e3-fd78-495f-99d9-98df1a3d3dca","Type":"ContainerDied","Data":"07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51"}
Jul 17 22:13:32.709537 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:32.709408    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07912b52ef035bab3a30ec64fe2b5f0fc531114e3dd78b6e9d481a67baeeed51"
Jul 17 22:13:36.089886 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.089856    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090156    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090166    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090180    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090186    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090193    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0dd740e3-fd78-495f-99d9-98df1a3d3dca" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090199    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="0dd740e3-fd78-495f-99d9-98df1a3d3dca" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090211    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090216    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090223    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090228    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090275    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090282    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ee3abb0-e5fe-4dbc-ab34-ac28f83c27a1" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090288    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="0dd740e3-fd78-495f-99d9-98df1a3d3dca" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090296    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.090349 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.090302    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="22978207-3f58-4c33-876c-e032df6d4420" containerName="revoke-keys"
Jul 17 22:13:36.093195 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.093176    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:36.095604 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.095570    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:13:36.099792 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.099765    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:36.232969 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.232938    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk\" (UID: \"89cc5f8f-1776-42e1-8512-baff4ad34b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:36.333694 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.333661    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk\" (UID: \"89cc5f8f-1776-42e1-8512-baff4ad34b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:36.334319 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.334301    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk\" (UID: \"89cc5f8f-1776-42e1-8512-baff4ad34b31\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:36.403974 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.403917    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:36.741199 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:36.741178    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:36.742402 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:36.742376    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89cc5f8f_1776_42e1_8512_baff4ad34b31.slice/crio-490b5defa5e02f83e4f39b091a0bf2de42311767957f1bd46ea4be9a631b48da WatchSource:0}: Error finding container 490b5defa5e02f83e4f39b091a0bf2de42311767957f1bd46ea4be9a631b48da: Status 404 returned error can't find the container with id 490b5defa5e02f83e4f39b091a0bf2de42311767957f1bd46ea4be9a631b48da
Jul 17 22:13:37.737736 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:37.737691    2546 generic.go:356] "Generic (PLEG): container finished" podID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerID="d9a588ac5e037a22d0e9962adb6c74b71a0918a3bb7b500284651057a35e14ae" exitCode=6
Jul 17 22:13:37.738173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:37.737778    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerDied","Data":"d9a588ac5e037a22d0e9962adb6c74b71a0918a3bb7b500284651057a35e14ae"}
Jul 17 22:13:37.738173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:37.737849    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerStarted","Data":"490b5defa5e02f83e4f39b091a0bf2de42311767957f1bd46ea4be9a631b48da"}
Jul 17 22:13:37.738173 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:37.737959    2546 scope.go:117] "RemoveContainer" containerID="d9a588ac5e037a22d0e9962adb6c74b71a0918a3bb7b500284651057a35e14ae"
Jul 17 22:13:38.745409 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:38.745377    2546 generic.go:356] "Generic (PLEG): container finished" podID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerID="93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd" exitCode=6
Jul 17 22:13:38.745898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:38.745454    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerDied","Data":"93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd"}
Jul 17 22:13:38.745898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:38.745512    2546 scope.go:117] "RemoveContainer" containerID="d9a588ac5e037a22d0e9962adb6c74b71a0918a3bb7b500284651057a35e14ae"
Jul 17 22:13:38.745898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:38.745700    2546 scope.go:117] "RemoveContainer" containerID="93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd"
Jul 17 22:13:38.746055 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:38.745947    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk_odh-ai-gateway-infra(89cc5f8f-1776-42e1-8512-baff4ad34b31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31"
Jul 17 22:13:39.752861 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:39.752833    2546 scope.go:117] "RemoveContainer" containerID="93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd"
Jul 17 22:13:39.753229 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:39.753033    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk_odh-ai-gateway-infra(89cc5f8f-1776-42e1-8512-baff4ad34b31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31"
Jul 17 22:13:41.282810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.282771    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:41.287281 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.287264    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:41.293110 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.293079    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:41.371397 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.371364    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs\" (UID: \"43965ed7-5ac2-4422-b2ff-a33e72c9bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:41.472301 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.472277    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs\" (UID: \"43965ed7-5ac2-4422-b2ff-a33e72c9bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:41.472852 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.472836    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs\" (UID: \"43965ed7-5ac2-4422-b2ff-a33e72c9bcef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:41.600523 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.600468    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:41.940364 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:41.940341    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:41.941971 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:41.941938    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43965ed7_5ac2_4422_b2ff_a33e72c9bcef.slice/crio-b2aaba542d4320c53a6d0b5ebda1b7442ad0e2f1382350a0abf3253624283e5a WatchSource:0}: Error finding container b2aaba542d4320c53a6d0b5ebda1b7442ad0e2f1382350a0abf3253624283e5a: Status 404 returned error can't find the container with id b2aaba542d4320c53a6d0b5ebda1b7442ad0e2f1382350a0abf3253624283e5a
Jul 17 22:13:42.769661 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:42.769629    2546 generic.go:356] "Generic (PLEG): container finished" podID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerID="ca475d61503f95d75006d9bca2272239e3214e34d92efcb32ba12159c61359bd" exitCode=6
Jul 17 22:13:42.770126 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:42.769687    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerDied","Data":"ca475d61503f95d75006d9bca2272239e3214e34d92efcb32ba12159c61359bd"}
Jul 17 22:13:42.770126 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:42.769767    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerStarted","Data":"b2aaba542d4320c53a6d0b5ebda1b7442ad0e2f1382350a0abf3253624283e5a"}
Jul 17 22:13:42.770126 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:42.769984    2546 scope.go:117] "RemoveContainer" containerID="ca475d61503f95d75006d9bca2272239e3214e34d92efcb32ba12159c61359bd"
Jul 17 22:13:43.777561 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:43.777528    2546 generic.go:356] "Generic (PLEG): container finished" podID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerID="af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c" exitCode=6
Jul 17 22:13:43.778007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:43.777628    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerDied","Data":"af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c"}
Jul 17 22:13:43.778007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:43.777690    2546 scope.go:117] "RemoveContainer" containerID="ca475d61503f95d75006d9bca2272239e3214e34d92efcb32ba12159c61359bd"
Jul 17 22:13:43.778007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:43.777807    2546 scope.go:117] "RemoveContainer" containerID="af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c"
Jul 17 22:13:43.778170 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:43.778007    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs_odh-ai-gateway-infra(43965ed7-5ac2-4422-b2ff-a33e72c9bcef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef"
Jul 17 22:13:44.785144 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:44.785115    2546 scope.go:117] "RemoveContainer" containerID="af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c"
Jul 17 22:13:44.785555 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:44.785285    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs_odh-ai-gateway-infra(43965ed7-5ac2-4422-b2ff-a33e72c9bcef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef"
Jul 17 22:13:53.531064 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:53.531025    2546 scope.go:117] "RemoveContainer" containerID="93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd"
Jul 17 22:13:53.836919 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:53.836893    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerStarted","Data":"03a582436eb11525451931ff66e2f9363b2a0373031ecff73f9c6c9865dc3504"}
Jul 17 22:13:53.855685 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:53.855638    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" podStartSLOduration=17.855620163 podStartE2EDuration="17.855620163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:53.852571022 +0000 UTC m=+3409.832941499" watchObservedRunningTime="2026-07-17 22:13:53.855620163 +0000 UTC m=+3409.835990644"
Jul 17 22:13:54.844786 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:54.844755    2546 generic.go:356] "Generic (PLEG): container finished" podID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerID="03a582436eb11525451931ff66e2f9363b2a0373031ecff73f9c6c9865dc3504" exitCode=6
Jul 17 22:13:54.845225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:54.844834    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerDied","Data":"03a582436eb11525451931ff66e2f9363b2a0373031ecff73f9c6c9865dc3504"}
Jul 17 22:13:54.845225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:54.844907    2546 scope.go:117] "RemoveContainer" containerID="93a6b39e8ed99f07822bf1acf37c150ab91b34e7c69fbbc4abbc3c023b2510bd"
Jul 17 22:13:54.845225 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:54.845217    2546 scope.go:117] "RemoveContainer" containerID="03a582436eb11525451931ff66e2f9363b2a0373031ecff73f9c6c9865dc3504"
Jul 17 22:13:54.845424 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:54.845404    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk_odh-ai-gateway-infra(89cc5f8f-1776-42e1-8512-baff4ad34b31)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31"
Jul 17 22:13:54.865617 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:54.865588    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:55.016551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.016525    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:13:55.016766 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.016736    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy" containerID="cri-o://30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7" gracePeriod=30
Jul 17 22:13:55.379316 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.379280    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"]
Jul 17 22:13:55.383663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.383642    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:55.390408 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.390385    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"]
Jul 17 22:13:55.483007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.482984    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp\" (UID: \"fe772978-5a63-40dd-8faa-9284146e65bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:55.583958 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.583932    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp\" (UID: \"fe772978-5a63-40dd-8faa-9284146e65bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:55.584519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.584497    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp\" (UID: \"fe772978-5a63-40dd-8faa-9284146e65bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:55.695241 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.695187    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:55.826464 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.826442    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"]
Jul 17 22:13:55.832388 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:55.832357    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe772978_5a63_40dd_8faa_9284146e65bb.slice/crio-170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc WatchSource:0}: Error finding container 170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc: Status 404 returned error can't find the container with id 170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc
Jul 17 22:13:55.853629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.853579    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp" event={"ID":"fe772978-5a63-40dd-8faa-9284146e65bb","Type":"ContainerStarted","Data":"170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc"}
Jul 17 22:13:55.988055 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:55.988034    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:56.086368 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.086345    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca\") pod \"89cc5f8f-1776-42e1-8512-baff4ad34b31\" (UID: \"89cc5f8f-1776-42e1-8512-baff4ad34b31\") "
Jul 17 22:13:56.086687 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.086664    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "89cc5f8f-1776-42e1-8512-baff4ad34b31" (UID: "89cc5f8f-1776-42e1-8512-baff4ad34b31"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:56.187167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.187146    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/89cc5f8f-1776-42e1-8512-baff4ad34b31-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:56.531189 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.531155    2546 scope.go:117] "RemoveContainer" containerID="af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c"
Jul 17 22:13:56.861109 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.861084    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"
Jul 17 22:13:56.861109 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.861091    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk" event={"ID":"89cc5f8f-1776-42e1-8512-baff4ad34b31","Type":"ContainerDied","Data":"490b5defa5e02f83e4f39b091a0bf2de42311767957f1bd46ea4be9a631b48da"}
Jul 17 22:13:56.861596 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.861157    2546 scope.go:117] "RemoveContainer" containerID="03a582436eb11525451931ff66e2f9363b2a0373031ecff73f9c6c9865dc3504"
Jul 17 22:13:56.863273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.863247    2546 generic.go:356] "Generic (PLEG): container finished" podID="fe772978-5a63-40dd-8faa-9284146e65bb" containerID="e7e0a07ba92e522ba892f3abbbf4c4d2f08b1172f2bc1c0cfe4ce839655c8058" exitCode=0
Jul 17 22:13:56.863933 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.863326    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp" event={"ID":"fe772978-5a63-40dd-8faa-9284146e65bb","Type":"ContainerDied","Data":"e7e0a07ba92e522ba892f3abbbf4c4d2f08b1172f2bc1c0cfe4ce839655c8058"}
Jul 17 22:13:56.865567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.865549    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerStarted","Data":"5ff5c99f1e5af34a710ff8811d4c689ba16632a6eb93a17e17fd54f7a2d4c752"}
Jul 17 22:13:56.877999 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.877978    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:56.885332 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.885310    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-hrsxk"]
Jul 17 22:13:56.912203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:56.912169    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" podStartSLOduration=15.912157548 podStartE2EDuration="15.912157548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:13:56.911175904 +0000 UTC m=+3412.891546382" watchObservedRunningTime="2026-07-17 22:13:56.912157548 +0000 UTC m=+3412.892528025"
Jul 17 22:13:57.874254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:57.874220    2546 generic.go:356] "Generic (PLEG): container finished" podID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerID="5ff5c99f1e5af34a710ff8811d4c689ba16632a6eb93a17e17fd54f7a2d4c752" exitCode=6
Jul 17 22:13:57.874679 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:57.874295    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerDied","Data":"5ff5c99f1e5af34a710ff8811d4c689ba16632a6eb93a17e17fd54f7a2d4c752"}
Jul 17 22:13:57.874679 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:57.874361    2546 scope.go:117] "RemoveContainer" containerID="af517cae285df5e1e79bb80dc63d554de429a2e5cd4e411e984a05a42ba3646c"
Jul 17 22:13:57.874679 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:57.874642    2546 scope.go:117] "RemoveContainer" containerID="5ff5c99f1e5af34a710ff8811d4c689ba16632a6eb93a17e17fd54f7a2d4c752"
Jul 17 22:13:57.875226 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:13:57.874878    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs_odh-ai-gateway-infra(43965ed7-5ac2-4422-b2ff-a33e72c9bcef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef"
Jul 17 22:13:57.922682 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:57.922642    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:58.006014 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.005987    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:58.100595 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.100565    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca\") pod \"fe772978-5a63-40dd-8faa-9284146e65bb\" (UID: \"fe772978-5a63-40dd-8faa-9284146e65bb\") "
Jul 17 22:13:58.100941 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.100921    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe772978-5a63-40dd-8faa-9284146e65bb" (UID: "fe772978-5a63-40dd-8faa-9284146e65bb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:58.201646 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.201574    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe772978-5a63-40dd-8faa-9284146e65bb-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:58.536089 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.536056    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" path="/var/lib/kubelet/pods/89cc5f8f-1776-42e1-8512-baff4ad34b31/volumes"
Jul 17 22:13:58.881828 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.881750    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp"
Jul 17 22:13:58.881828 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.881766    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp" event={"ID":"fe772978-5a63-40dd-8faa-9284146e65bb","Type":"ContainerDied","Data":"170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc"}
Jul 17 22:13:58.882279 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:58.881828    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170a0951fbe71adbe622e640e489ed07dbf20d6d36c0210dda33e6473325c2fc"
Jul 17 22:13:59.019449 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.019425    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:59.110713 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.110681    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca\") pod \"43965ed7-5ac2-4422-b2ff-a33e72c9bcef\" (UID: \"43965ed7-5ac2-4422-b2ff-a33e72c9bcef\") "
Jul 17 22:13:59.111012 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.110992    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43965ed7-5ac2-4422-b2ff-a33e72c9bcef" (UID: "43965ed7-5ac2-4422-b2ff-a33e72c9bcef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:13:59.212091 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.212027    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43965ed7-5ac2-4422-b2ff-a33e72c9bcef-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:13:59.520829 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.520798    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:13:59.521121 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521109    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521123    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521134    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe772978-5a63-40dd-8faa-9284146e65bb" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521139    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe772978-5a63-40dd-8faa-9284146e65bb" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521146    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521151    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521161    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521166    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521172    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521177    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521185    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521190    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521243    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521251    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521256    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521263    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:13:59.521419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.521268    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe772978-5a63-40dd-8faa-9284146e65bb" containerName="revoke-keys"
Jul 17 22:13:59.524162 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.524146    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:13:59.530852 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.530825    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:13:59.615644 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.615614    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc\" (UID: \"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:13:59.716936 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.716910    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc\" (UID: \"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:13:59.717492 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.717474    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc\" (UID: \"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:13:59.834996 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.834942    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:13:59.893388 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.893361    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"
Jul 17 22:13:59.893826 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.893354    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs" event={"ID":"43965ed7-5ac2-4422-b2ff-a33e72c9bcef","Type":"ContainerDied","Data":"b2aaba542d4320c53a6d0b5ebda1b7442ad0e2f1382350a0abf3253624283e5a"}
Jul 17 22:13:59.893826 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.893514    2546 scope.go:117] "RemoveContainer" containerID="5ff5c99f1e5af34a710ff8811d4c689ba16632a6eb93a17e17fd54f7a2d4c752"
Jul 17 22:13:59.919415 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.919391    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:59.925183 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.925142    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-jrxfs"]
Jul 17 22:13:59.971676 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:13:59.971651    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:13:59.973444 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:13:59.973405    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a82a7e9_7718_4826_a7c3_71ba4f7fbc19.slice/crio-e69af62927adfb7ccbd933cf4c443390dbb71268b515243714fce431ce838fb2 WatchSource:0}: Error finding container e69af62927adfb7ccbd933cf4c443390dbb71268b515243714fce431ce838fb2: Status 404 returned error can't find the container with id e69af62927adfb7ccbd933cf4c443390dbb71268b515243714fce431ce838fb2
Jul 17 22:14:00.260167 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.260146    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:14:00.423046 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.422990    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423046 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423030    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423064    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423089    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423126    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423149    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423222 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423175    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423285    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423344    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-95zd9\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9\") pod \"9b663148-c97e-43be-a3d1-fb6183f66014\" (UID: \"9b663148-c97e-43be-a3d1-fb6183f66014\") "
Jul 17 22:14:00.423481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423441    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:00.423481 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423447    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:00.423714 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423543    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:00.423823 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423712    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:00.423823 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423736    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.423937 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423859    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data" (OuterVolumeSpecName: "istio-data") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:00.423937 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423894    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9b663148-c97e-43be-a3d1-fb6183f66014-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.423937 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.423912    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.458097 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.458066    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:14:00.458318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.458109    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:00.458318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.458137    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:14:00.458318 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.458152    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9" (OuterVolumeSpecName: "kube-api-access-95zd9") pod "9b663148-c97e-43be-a3d1-fb6183f66014" (UID: "9b663148-c97e-43be-a3d1-fb6183f66014"). InnerVolumeSpecName "kube-api-access-95zd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:14:00.525204 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525185    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.525204 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525203    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.525334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525213    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-95zd9\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-kube-api-access-95zd9\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.525334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525221    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9b663148-c97e-43be-a3d1-fb6183f66014-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.525334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525230    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9b663148-c97e-43be-a3d1-fb6183f66014-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.525334 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.525238    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b663148-c97e-43be-a3d1-fb6183f66014-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:00.536518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.536497    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" path="/var/lib/kubelet/pods/43965ed7-5ac2-4422-b2ff-a33e72c9bcef/volumes"
Jul 17 22:14:00.900276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.900236    2546 generic.go:356] "Generic (PLEG): container finished" podID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerID="495ac993c28ead1cfe1ed639152663ce6dd215a804c0b34a00d1be025e71a06c" exitCode=6
Jul 17 22:14:00.900677 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.900325    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerDied","Data":"495ac993c28ead1cfe1ed639152663ce6dd215a804c0b34a00d1be025e71a06c"}
Jul 17 22:14:00.900677 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.900382    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerStarted","Data":"e69af62927adfb7ccbd933cf4c443390dbb71268b515243714fce431ce838fb2"}
Jul 17 22:14:00.900677 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.900511    2546 scope.go:117] "RemoveContainer" containerID="495ac993c28ead1cfe1ed639152663ce6dd215a804c0b34a00d1be025e71a06c"
Jul 17 22:14:00.902344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.902325    2546 generic.go:356] "Generic (PLEG): container finished" podID="9b663148-c97e-43be-a3d1-fb6183f66014" containerID="30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7" exitCode=0
Jul 17 22:14:00.902442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.902366    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"
Jul 17 22:14:00.902442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.902396    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerDied","Data":"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"}
Jul 17 22:14:00.902442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.902437    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v" event={"ID":"9b663148-c97e-43be-a3d1-fb6183f66014","Type":"ContainerDied","Data":"833b24d2d01d3909e32ca4c509469b431f770a2fdf2a17d6beff7dab9855f00e"}
Jul 17 22:14:00.902606 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.902459    2546 scope.go:117] "RemoveContainer" containerID="30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"
Jul 17 22:14:00.912545 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.912528    2546 scope.go:117] "RemoveContainer" containerID="f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"
Jul 17 22:14:00.924259 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.924238    2546 scope.go:117] "RemoveContainer" containerID="30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"
Jul 17 22:14:00.924635 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:00.924608    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7\": container with ID starting with 30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7 not found: ID does not exist" containerID="30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"
Jul 17 22:14:00.924810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.924646    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7"} err="failed to get container status \"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7\": rpc error: code = NotFound desc = could not find container \"30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7\": container with ID starting with 30160a33e03fbf5ca333ca93b8e5e96ba95874d8ce77ffc7faec3328f39422b7 not found: ID does not exist"
Jul 17 22:14:00.924810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.924670    2546 scope.go:117] "RemoveContainer" containerID="f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"
Jul 17 22:14:00.925358 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:00.925250    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620\": container with ID starting with f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620 not found: ID does not exist" containerID="f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"
Jul 17 22:14:00.925506 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.925478    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620"} err="failed to get container status \"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620\": rpc error: code = NotFound desc = could not find container \"f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620\": container with ID starting with f6c344a4c5d29b7b24c920e44bfff736a2fcf8a91b0d260aba339bdff3faf620 not found: ID does not exist"
Jul 17 22:14:00.927570 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.927549    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:14:00.930825 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:00.930806    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-2a645a-openshift-default-5fbb668b9f-k597v"]
Jul 17 22:14:01.910440 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:01.910403    2546 generic.go:356] "Generic (PLEG): container finished" podID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerID="bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91" exitCode=6
Jul 17 22:14:01.910951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:01.910484    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerDied","Data":"bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91"}
Jul 17 22:14:01.910951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:01.910579    2546 scope.go:117] "RemoveContainer" containerID="495ac993c28ead1cfe1ed639152663ce6dd215a804c0b34a00d1be025e71a06c"
Jul 17 22:14:01.910951 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:01.910825    2546 scope.go:117] "RemoveContainer" containerID="bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91"
Jul 17 22:14:01.911103 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:01.911024    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc_odh-ai-gateway-infra(8a82a7e9-7718-4826-a7c3-71ba4f7fbc19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19"
Jul 17 22:14:02.537400 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:02.537369    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" path="/var/lib/kubelet/pods/9b663148-c97e-43be-a3d1-fb6183f66014/volumes"
Jul 17 22:14:02.918732 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:02.918633    2546 scope.go:117] "RemoveContainer" containerID="bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91"
Jul 17 22:14:02.919100 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:02.918821    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc_odh-ai-gateway-infra(8a82a7e9-7718-4826-a7c3-71ba4f7fbc19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19"
Jul 17 22:14:08.649603 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.649560    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.649907    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.649924    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.649943    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.649948    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.650003    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="89cc5f8f-1776-42e1-8512-baff4ad34b31" containerName="revoke-keys"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.650012    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.650019    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:08.650153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.650024    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="43965ed7-5ac2-4422-b2ff-a33e72c9bcef" containerName="revoke-keys"
Jul 17 22:14:08.655045 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.655024    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:08.659136 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.659103    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:08.792292 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.792256    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d\" (UID: \"423e7e54-a4a3-4c1e-9385-c2e96818d37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:08.893051 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.893021    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d\" (UID: \"423e7e54-a4a3-4c1e-9385-c2e96818d37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:08.893628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.893609    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d\" (UID: \"423e7e54-a4a3-4c1e-9385-c2e96818d37c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:08.967685 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:08.967626    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:09.098124 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:09.098109    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:09.100985 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:14:09.100950    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423e7e54_a4a3_4c1e_9385_c2e96818d37c.slice/crio-7cf71b095bd9754e36d34f64cb6d180c1941e8d5f481e6edfbdbfa640d76e7b7 WatchSource:0}: Error finding container 7cf71b095bd9754e36d34f64cb6d180c1941e8d5f481e6edfbdbfa640d76e7b7: Status 404 returned error can't find the container with id 7cf71b095bd9754e36d34f64cb6d180c1941e8d5f481e6edfbdbfa640d76e7b7
Jul 17 22:14:09.957309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:09.957229    2546 generic.go:356] "Generic (PLEG): container finished" podID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerID="0204f3be1b8af001437e8439b0680b169018713bb37c3bc42457f9d4b71024b8" exitCode=6
Jul 17 22:14:09.957674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:09.957316    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerDied","Data":"0204f3be1b8af001437e8439b0680b169018713bb37c3bc42457f9d4b71024b8"}
Jul 17 22:14:09.957674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:09.957376    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerStarted","Data":"7cf71b095bd9754e36d34f64cb6d180c1941e8d5f481e6edfbdbfa640d76e7b7"}
Jul 17 22:14:09.957674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:09.957496    2546 scope.go:117] "RemoveContainer" containerID="0204f3be1b8af001437e8439b0680b169018713bb37c3bc42457f9d4b71024b8"
Jul 17 22:14:10.964791 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:10.964686    2546 generic.go:356] "Generic (PLEG): container finished" podID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerID="0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d" exitCode=6
Jul 17 22:14:10.964791 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:10.964771    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerDied","Data":"0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d"}
Jul 17 22:14:10.965295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:10.964837    2546 scope.go:117] "RemoveContainer" containerID="0204f3be1b8af001437e8439b0680b169018713bb37c3bc42457f9d4b71024b8"
Jul 17 22:14:10.965295 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:10.964996    2546 scope.go:117] "RemoveContainer" containerID="0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d"
Jul 17 22:14:10.965295 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:10.965247    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d_odh-ai-gateway-infra(423e7e54-a4a3-4c1e-9385-c2e96818d37c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c"
Jul 17 22:14:11.972954 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:11.972923    2546 scope.go:117] "RemoveContainer" containerID="0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d"
Jul 17 22:14:11.973367 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:11.973129    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d_odh-ai-gateway-infra(423e7e54-a4a3-4c1e-9385-c2e96818d37c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c"
Jul 17 22:14:15.531491 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:15.531461    2546 scope.go:117] "RemoveContainer" containerID="bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91"
Jul 17 22:14:15.995170 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:15.995139    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerStarted","Data":"cb320cc3046daab539af292464d43fc2637f9eeb282fc5dd4665556abf3c24e8"}
Jul 17 22:14:16.016462 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:16.016416    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" podStartSLOduration=17.016402699 podStartE2EDuration="17.016402699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:13:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:16.015332134 +0000 UTC m=+3431.995702615" watchObservedRunningTime="2026-07-17 22:14:16.016402699 +0000 UTC m=+3431.996773175"
Jul 17 22:14:17.002186 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:17.002152    2546 generic.go:356] "Generic (PLEG): container finished" podID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerID="cb320cc3046daab539af292464d43fc2637f9eeb282fc5dd4665556abf3c24e8" exitCode=6
Jul 17 22:14:17.002620 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:17.002221    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerDied","Data":"cb320cc3046daab539af292464d43fc2637f9eeb282fc5dd4665556abf3c24e8"}
Jul 17 22:14:17.002620 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:17.002280    2546 scope.go:117] "RemoveContainer" containerID="bb4a993839ad437628f324e460c3957f16a33eb9e45205bd9ae16a97432d1b91"
Jul 17 22:14:17.002620 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:17.002525    2546 scope.go:117] "RemoveContainer" containerID="cb320cc3046daab539af292464d43fc2637f9eeb282fc5dd4665556abf3c24e8"
Jul 17 22:14:17.002757 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:17.002737    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc_odh-ai-gateway-infra(8a82a7e9-7718-4826-a7c3-71ba4f7fbc19)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19"
Jul 17 22:14:17.044228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:17.044193    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:14:18.141325 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:18.141299    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:14:18.154967 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:18.154949    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca\") pod \"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19\" (UID: \"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19\") "
Jul 17 22:14:18.155264 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:18.155244    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" (UID: "8a82a7e9-7718-4826-a7c3-71ba4f7fbc19"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:18.255636 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:18.255612    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:19.015483 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:19.015450    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"
Jul 17 22:14:19.015483 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:19.015467    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc" event={"ID":"8a82a7e9-7718-4826-a7c3-71ba4f7fbc19","Type":"ContainerDied","Data":"e69af62927adfb7ccbd933cf4c443390dbb71268b515243714fce431ce838fb2"}
Jul 17 22:14:19.015797 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:19.015556    2546 scope.go:117] "RemoveContainer" containerID="cb320cc3046daab539af292464d43fc2637f9eeb282fc5dd4665556abf3c24e8"
Jul 17 22:14:19.034031 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:19.034007    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:14:19.037270 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:19.037245    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-nwbwc"]
Jul 17 22:14:20.536666 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:20.536630    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" path="/var/lib/kubelet/pods/8a82a7e9-7718-4826-a7c3-71ba4f7fbc19/volumes"
Jul 17 22:14:21.120073 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:21.120042    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:14:21.120269 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:21.120246    2546 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy" containerID="cri-o://b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68" gracePeriod=30
Jul 17 22:14:26.379909 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.379887    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:14:26.413890 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.413829    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414025 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.413948    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414025 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.413987    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-67xlk\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414025 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414012    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414057    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:26.414198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414107    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414152    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414198 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414187    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414401 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414216    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414401 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414255    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs\") pod \"213f2544-8a42-406f-9f13-792c711d736e\" (UID: \"213f2544-8a42-406f-9f13-792c711d736e\") "
Jul 17 22:14:26.414401 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414331    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:26.414635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414514    2546 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-credential-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.414635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414532    2546 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/213f2544-8a42-406f-9f13-792c711d736e-istiod-ca-cert\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.414635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414561    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:26.414842 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.414784    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:26.415355 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.415336    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data" (OuterVolumeSpecName: "istio-data") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:26.416403 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.416377    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 17 22:14:26.416772 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.416749    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 17 22:14:26.416772 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.416759    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk" (OuterVolumeSpecName: "kube-api-access-67xlk") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "kube-api-access-67xlk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:14:26.416942 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.416771    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token" (OuterVolumeSpecName: "istio-token") pod "213f2544-8a42-406f-9f13-792c711d736e" (UID: "213f2544-8a42-406f-9f13-792c711d736e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 17 22:14:26.514963 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514941    2546 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67xlk\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-kube-api-access-67xlk\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.514963 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514961    2546 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/213f2544-8a42-406f-9f13-792c711d736e-istio-token\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.515088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514971    2546 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-envoy\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.515088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514979    2546 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-socket\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.515088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514987    2546 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/213f2544-8a42-406f-9f13-792c711d736e-istio-podinfo\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.515088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.514994    2546 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-istio-data\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.515088 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.515002    2546 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/213f2544-8a42-406f-9f13-792c711d736e-workload-certs\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:26.531664 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:26.531639    2546 scope.go:117] "RemoveContainer" containerID="0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d"
Jul 17 22:14:27.067455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.067378    2546 generic.go:356] "Generic (PLEG): container finished" podID="213f2544-8a42-406f-9f13-792c711d736e" containerID="b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68" exitCode=0
Jul 17 22:14:27.067455 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.067427    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"
Jul 17 22:14:27.067684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.067460    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerDied","Data":"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"}
Jul 17 22:14:27.067684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.067507    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws" event={"ID":"213f2544-8a42-406f-9f13-792c711d736e","Type":"ContainerDied","Data":"0ee6aa55b90afd2f38012a6115794e6b68a341698b55cef015293d739106e629"}
Jul 17 22:14:27.067684 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.067532    2546 scope.go:117] "RemoveContainer" containerID="b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"
Jul 17 22:14:27.069618 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.069595    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerStarted","Data":"07a0480059954087bd0e0b0ff11c15a0898605474765f976836a1116b5057180"}
Jul 17 22:14:27.076354 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.076342    2546 scope.go:117] "RemoveContainer" containerID="41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"
Jul 17 22:14:27.084280 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.084263    2546 scope.go:117] "RemoveContainer" containerID="b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"
Jul 17 22:14:27.084517 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:27.084499    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68\": container with ID starting with b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68 not found: ID does not exist" containerID="b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"
Jul 17 22:14:27.084592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.084524    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68"} err="failed to get container status \"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68\": rpc error: code = NotFound desc = could not find container \"b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68\": container with ID starting with b139bc85e18938c11607a8746276b36657d18a8d0d9f24022884c1774bbb5a68 not found: ID does not exist"
Jul 17 22:14:27.084592 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.084541    2546 scope.go:117] "RemoveContainer" containerID="41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"
Jul 17 22:14:27.084791 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:27.084773    2546 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad\": container with ID starting with 41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad not found: ID does not exist" containerID="41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"
Jul 17 22:14:27.084866 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.084795    2546 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad"} err="failed to get container status \"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad\": rpc error: code = NotFound desc = could not find container \"41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad\": container with ID starting with 41eb300bddabcb29a566a22f24abf641c38fd96c37bac6eed3b50c5d5e54bbad not found: ID does not exist"
Jul 17 22:14:27.086276 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.086254    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:14:27.090780 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.090761    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b96f25-openshift-default-667b97dd87-2rdws"]
Jul 17 22:14:27.101192 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.101147    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" podStartSLOduration=19.101130529 podStartE2EDuration="19.101130529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:27.100556466 +0000 UTC m=+3443.080926944" watchObservedRunningTime="2026-07-17 22:14:27.101130529 +0000 UTC m=+3443.081501006"
Jul 17 22:14:27.564284 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.564253    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6dbd9688-wf99g_789287cc-65cb-4d08-9c36-320a8146443e/manager/0.log"
Jul 17 22:14:27.570350 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:27.570326    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6dbd9688-wf99g_789287cc-65cb-4d08-9c36-320a8146443e/copy-manifests/0.log"
Jul 17 22:14:28.076874 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.076848    2546 generic.go:356] "Generic (PLEG): container finished" podID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerID="07a0480059954087bd0e0b0ff11c15a0898605474765f976836a1116b5057180" exitCode=6
Jul 17 22:14:28.077032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.076919    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerDied","Data":"07a0480059954087bd0e0b0ff11c15a0898605474765f976836a1116b5057180"}
Jul 17 22:14:28.077032 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.076975    2546 scope.go:117] "RemoveContainer" containerID="0f9cae85ee7c451cf6744fb2db9565208abf6ddeb3020cedfd988fa0f3fea15d"
Jul 17 22:14:28.077254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.077228    2546 scope.go:117] "RemoveContainer" containerID="07a0480059954087bd0e0b0ff11c15a0898605474765f976836a1116b5057180"
Jul 17 22:14:28.077429 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:28.077401    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d_odh-ai-gateway-infra(423e7e54-a4a3-4c1e-9385-c2e96818d37c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c"
Jul 17 22:14:28.112695 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.112663    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:28.132798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.132782    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738730-j4h9q_263b66e2-e7e7-4393-9269-5aad90490125/cleanup/0.log"
Jul 17 22:14:28.238184 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.238150    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738745-sl745_a21fc198-48e6-4e00-96c7-c1a9257af5cf/cleanup/0.log"
Jul 17 22:14:28.347231 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.347171    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29738760-rn7hp_e240e5d5-feee-47df-a2b0-e8fb59ada5a5/cleanup/0.log"
Jul 17 22:14:28.451643 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.451622    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d_423e7e54-a4a3-4c1e-9385-c2e96818d37c/revoke-keys/2.log"
Jul 17 22:14:28.536464 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.536435    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213f2544-8a42-406f-9f13-792c711d736e" path="/var/lib/kubelet/pods/213f2544-8a42-406f-9f13-792c711d736e/volumes"
Jul 17 22:14:28.556430 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.556408    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-e12150-95t2q_771d4a38-47e9-4b4d-b59b-02dafd38e12a/revoke-keys/0.log"
Jul 17 22:14:28.677559 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.677502    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-92c219-wld6x_0feed117-bc75-4c3e-9317-bd86a7a79983/revoke-keys/0.log"
Jul 17 22:14:28.786599 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.786572    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-592f58-lk8hp_b6a9f1fb-b6ab-4dc6-9b92-c9b4370407ed/revoke-keys/0.log"
Jul 17 22:14:28.889861 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.889837    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-6be22b-zh4d8_711b4a98-240f-43bc-ad9e-25909b9c4d6c/revoke-keys/0.log"
Jul 17 22:14:28.999529 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:28.999504    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-479788cb-vszss_285d910c-3ec2-4e95-9459-637ef0fa2144/revoke-keys/0.log"
Jul 17 22:14:29.107142 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.107116    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-6e20ff19-t7dph_7bbbc586-57f2-4f36-82dd-d5a0694b37b0/revoke-keys/0.log"
Jul 17 22:14:29.210135 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.210117    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-a-2a645a-5f94t_0dd740e3-fd78-495f-99d9-98df1a3d3dca/revoke-keys/0.log"
Jul 17 22:14:29.212309 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.212291    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:29.235831 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.235807    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca\") pod \"423e7e54-a4a3-4c1e-9385-c2e96818d37c\" (UID: \"423e7e54-a4a3-4c1e-9385-c2e96818d37c\") "
Jul 17 22:14:29.236095 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.236076    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423e7e54-a4a3-4c1e-9385-c2e96818d37c" (UID: "423e7e54-a4a3-4c1e-9385-c2e96818d37c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:29.317923 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.317844    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-b96f25-cg6mp_fe772978-5a63-40dd-8faa-9284146e65bb/revoke-keys/0.log"
Jul 17 22:14:29.337425 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.337405    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423e7e54-a4a3-4c1e-9385-c2e96818d37c-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:29.424552 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:29.424528    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-gz6fk_7f6fb0ce-2931-4750-ab12-ca50960201bb/postgres/0.log"
Jul 17 22:14:30.099201 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.099167    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"
Jul 17 22:14:30.099689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.099166    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d" event={"ID":"423e7e54-a4a3-4c1e-9385-c2e96818d37c","Type":"ContainerDied","Data":"7cf71b095bd9754e36d34f64cb6d180c1941e8d5f481e6edfbdbfa640d76e7b7"}
Jul 17 22:14:30.099689 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.099369    2546 scope.go:117] "RemoveContainer" containerID="07a0480059954087bd0e0b0ff11c15a0898605474765f976836a1116b5057180"
Jul 17 22:14:30.121400 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.121368    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:30.126956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.126934    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-vbm6d"]
Jul 17 22:14:30.163182 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.163152    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/util/0.log"
Jul 17 22:14:30.169795 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.169778    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/pull/0.log"
Jul 17 22:14:30.175675 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.175661    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/extract/0.log"
Jul 17 22:14:30.282602 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.282583    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/util/0.log"
Jul 17 22:14:30.288686 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.288666    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/pull/0.log"
Jul 17 22:14:30.295868 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.295851    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/extract/0.log"
Jul 17 22:14:30.402277 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.402206    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/pull/0.log"
Jul 17 22:14:30.408522 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.408507    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/extract/0.log"
Jul 17 22:14:30.414698 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.414680    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/util/0.log"
Jul 17 22:14:30.522449 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.522425    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/pull/0.log"
Jul 17 22:14:30.528899 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.528877    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/extract/0.log"
Jul 17 22:14:30.534502 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.534484    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/util/0.log"
Jul 17 22:14:30.536055 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:30.536035    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" path="/var/lib/kubelet/pods/423e7e54-a4a3-4c1e-9385-c2e96818d37c/volumes"
Jul 17 22:14:31.325668 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325632    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325936    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325948    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325956    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325961    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325970    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325976    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325983    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325987    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b663148-c97e-43be-a3d1-fb6183f66014" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325994    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.325999    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326008    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326013    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326018    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326023    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326029    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326034    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326043    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326048    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326094    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.326628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326101    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326107    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:31.326628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326113    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:31.326628 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.326120    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:31.330448 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.330431    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:31.332947 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.332804    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:14:31.336085 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.336061    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:31.351031 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.351009    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2\" (UID: \"6289ab0b-4098-4b37-9889-40e91ebb7a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:31.452038 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.452006    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2\" (UID: \"6289ab0b-4098-4b37-9889-40e91ebb7a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:31.452575 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.452557    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2\" (UID: \"6289ab0b-4098-4b37-9889-40e91ebb7a53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:31.642911 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.642843    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:31.770461 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.770441    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:31.772049 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:14:31.772025    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6289ab0b_4098_4b37_9889_40e91ebb7a53.slice/crio-81f06fa4f7ff5ad9f14057055e30b72c80532b54d136646d94ebc987799d87c1 WatchSource:0}: Error finding container 81f06fa4f7ff5ad9f14057055e30b72c80532b54d136646d94ebc987799d87c1: Status 404 returned error can't find the container with id 81f06fa4f7ff5ad9f14057055e30b72c80532b54d136646d94ebc987799d87c1
Jul 17 22:14:31.880209 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:31.880189    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gzv5k_ad5b0183-f8a6-4173-9c16-4a91202649b0/discovery/0.log"
Jul 17 22:14:32.111253 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:32.111221    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerStarted","Data":"ca5add3e5d6ddeecb1ab60c91f2850b18dad00a08c9610108aa6b80af984bb9f"}
Jul 17 22:14:32.111436 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:32.111276    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerStarted","Data":"81f06fa4f7ff5ad9f14057055e30b72c80532b54d136646d94ebc987799d87c1"}
Jul 17 22:14:32.126253 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:32.126212    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" podStartSLOduration=1.12619791 podStartE2EDuration="1.12619791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:32.125360336 +0000 UTC m=+3448.105730812" watchObservedRunningTime="2026-07-17 22:14:32.12619791 +0000 UTC m=+3448.106568386"
Jul 17 22:14:32.867151 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:32.867126    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-xv6s5_12b2e240-1d31-408f-87af-3b314f848141/main/0.log"
Jul 17 22:14:32.972623 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:32.972595    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-k4hr6_23a6217f-72e7-4b31-9847-d22f68c28332/main/0.log"
Jul 17 22:14:33.117402 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:33.117304    2546 generic.go:356] "Generic (PLEG): container finished" podID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerID="ca5add3e5d6ddeecb1ab60c91f2850b18dad00a08c9610108aa6b80af984bb9f" exitCode=6
Jul 17 22:14:33.117563 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:33.117404    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerDied","Data":"ca5add3e5d6ddeecb1ab60c91f2850b18dad00a08c9610108aa6b80af984bb9f"}
Jul 17 22:14:33.117563 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:33.117550    2546 scope.go:117] "RemoveContainer" containerID="ca5add3e5d6ddeecb1ab60c91f2850b18dad00a08c9610108aa6b80af984bb9f"
Jul 17 22:14:34.131228 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:34.131193    2546 generic.go:356] "Generic (PLEG): container finished" podID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerID="0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb" exitCode=6
Jul 17 22:14:34.131688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:34.131274    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerDied","Data":"0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb"}
Jul 17 22:14:34.131688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:34.131351    2546 scope.go:117] "RemoveContainer" containerID="ca5add3e5d6ddeecb1ab60c91f2850b18dad00a08c9610108aa6b80af984bb9f"
Jul 17 22:14:34.131688 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:34.131504    2546 scope.go:117] "RemoveContainer" containerID="0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb"
Jul 17 22:14:34.131869 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:34.131714    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2_odh-ai-gateway-infra(6289ab0b-4098-4b37-9889-40e91ebb7a53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53"
Jul 17 22:14:35.138096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:35.138068    2546 scope.go:117] "RemoveContainer" containerID="0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb"
Jul 17 22:14:35.138484 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:35.138242    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2_odh-ai-gateway-infra(6289ab0b-4098-4b37-9889-40e91ebb7a53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53"
Jul 17 22:14:36.135009 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.134977    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:36.135354 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.135340    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="423e7e54-a4a3-4c1e-9385-c2e96818d37c" containerName="revoke-keys"
Jul 17 22:14:36.135408 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.135357    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="213f2544-8a42-406f-9f13-792c711d736e" containerName="istio-proxy"
Jul 17 22:14:36.135408 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.135372    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="8a82a7e9-7718-4826-a7c3-71ba4f7fbc19" containerName="revoke-keys"
Jul 17 22:14:36.139534 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.139513    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:36.145798 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.145772    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:36.189450 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.189423    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh\" (UID: \"ebd648af-15c5-4f46-99e9-3bc9f8a3db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:36.289991 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.289966    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh\" (UID: \"ebd648af-15c5-4f46-99e9-3bc9f8a3db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:36.290535 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.290517    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh\" (UID: \"ebd648af-15c5-4f46-99e9-3bc9f8a3db16\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:36.451336 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.451276    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:36.585516 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:36.585488    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:36.587296 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:14:36.587271    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebd648af_15c5_4f46_99e9_3bc9f8a3db16.slice/crio-b90da46c0ded19ae8ab2007709059ee6ac6a7d490757c2333f77911540c1e103 WatchSource:0}: Error finding container b90da46c0ded19ae8ab2007709059ee6ac6a7d490757c2333f77911540c1e103: Status 404 returned error can't find the container with id b90da46c0ded19ae8ab2007709059ee6ac6a7d490757c2333f77911540c1e103
Jul 17 22:14:37.152026 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:37.151920    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerStarted","Data":"759e12986c30489b80ec7c4d68eb1eb5b88144f2d4046e841fbfef823025f707"}
Jul 17 22:14:37.152026 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:37.151989    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerStarted","Data":"b90da46c0ded19ae8ab2007709059ee6ac6a7d490757c2333f77911540c1e103"}
Jul 17 22:14:37.167632 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:37.167569    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" podStartSLOduration=1.1675550000000001 podStartE2EDuration="1.167555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:37.167241657 +0000 UTC m=+3453.147612135" watchObservedRunningTime="2026-07-17 22:14:37.167555 +0000 UTC m=+3453.147925474"
Jul 17 22:14:38.159152 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:38.159116    2546 generic.go:356] "Generic (PLEG): container finished" podID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerID="759e12986c30489b80ec7c4d68eb1eb5b88144f2d4046e841fbfef823025f707" exitCode=6
Jul 17 22:14:38.159663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:38.159200    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerDied","Data":"759e12986c30489b80ec7c4d68eb1eb5b88144f2d4046e841fbfef823025f707"}
Jul 17 22:14:38.159663 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:38.159402    2546 scope.go:117] "RemoveContainer" containerID="759e12986c30489b80ec7c4d68eb1eb5b88144f2d4046e841fbfef823025f707"
Jul 17 22:14:39.166788 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:39.166682    2546 generic.go:356] "Generic (PLEG): container finished" podID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerID="90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049" exitCode=6
Jul 17 22:14:39.166788 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:39.166755    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerDied","Data":"90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049"}
Jul 17 22:14:39.167248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:39.166807    2546 scope.go:117] "RemoveContainer" containerID="759e12986c30489b80ec7c4d68eb1eb5b88144f2d4046e841fbfef823025f707"
Jul 17 22:14:39.167248 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:39.166946    2546 scope.go:117] "RemoveContainer" containerID="90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049"
Jul 17 22:14:39.167248 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:39.167145    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh_odh-ai-gateway-infra(ebd648af-15c5-4f46-99e9-3bc9f8a3db16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16"
Jul 17 22:14:40.173232 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:40.173195    2546 scope.go:117] "RemoveContainer" containerID="90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049"
Jul 17 22:14:40.173848 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:40.173534    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh_odh-ai-gateway-infra(ebd648af-15c5-4f46-99e9-3bc9f8a3db16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16"
Jul 17 22:14:41.233274 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:41.233246    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mwjff_aff86450-cf63-48d6-8ceb-d720e646190d/global-pull-secret-syncer/0.log"
Jul 17 22:14:41.312387 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:41.312359    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-jsttb_c0620747-a811-4560-942d-1cb176ff1af5/konnectivity-agent/0.log"
Jul 17 22:14:41.418983 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:41.418951    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-135-220.ec2.internal_d3d415bcf690ad6d0665b9c29c5a5dbf/haproxy/0.log"
Jul 17 22:14:46.135907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.135875    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/extract/0.log"
Jul 17 22:14:46.161379 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.161350    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/util/0.log"
Jul 17 22:14:46.196699 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.196677    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b759cjwhn_d0ed2b81-1c86-4721-9ab5-d248cc4a053b/pull/0.log"
Jul 17 22:14:46.238427 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.238406    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/extract/0.log"
Jul 17 22:14:46.275378 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.275359    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/util/0.log"
Jul 17 22:14:46.316778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.316761    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e04fl7f_2c7df81c-55a3-4132-9247-a6befdd7928f/pull/0.log"
Jul 17 22:14:46.354578 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.354550    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/extract/0.log"
Jul 17 22:14:46.386579 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.386525    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/util/0.log"
Jul 17 22:14:46.409254 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.409236    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73t8wc2_5634c3de-f7cb-4da5-beff-bf4ccbc0215a/pull/0.log"
Jul 17 22:14:46.437164 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.437137    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/extract/0.log"
Jul 17 22:14:46.468630 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.468605    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/util/0.log"
Jul 17 22:14:46.492039 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:46.492017    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef19vflc_ee99d263-71ca-4bc9-b145-423943857726/pull/0.log"
Jul 17 22:14:48.531790 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:48.531766    2546 scope.go:117] "RemoveContainer" containerID="0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb"
Jul 17 22:14:49.221822 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.221795    2546 generic.go:356] "Generic (PLEG): container finished" podID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerID="e0093ee4f18a58bd563a57df06e9c60540a0554df57b0fcfd7e831c02b98e5ae" exitCode=6
Jul 17 22:14:49.221930 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.221864    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerDied","Data":"e0093ee4f18a58bd563a57df06e9c60540a0554df57b0fcfd7e831c02b98e5ae"}
Jul 17 22:14:49.221930 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.221918    2546 scope.go:117] "RemoveContainer" containerID="0099964d7220db7174cd645aeb2162a50d0e81b298ce1fc90b21e6fab7e8e0bb"
Jul 17 22:14:49.222203 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.222179    2546 scope.go:117] "RemoveContainer" containerID="e0093ee4f18a58bd563a57df06e9c60540a0554df57b0fcfd7e831c02b98e5ae"
Jul 17 22:14:49.222406 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:49.222384    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2_odh-ai-gateway-infra(6289ab0b-4098-4b37-9889-40e91ebb7a53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53"
Jul 17 22:14:49.415068 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.415014    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdr5c_a9a43db9-c576-4bd4-a25d-8af7ad92303c/node-exporter/0.log"
Jul 17 22:14:49.435454 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.435436    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdr5c_a9a43db9-c576-4bd4-a25d-8af7ad92303c/kube-rbac-proxy/0.log"
Jul 17 22:14:49.457778 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.457763    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-kdr5c_a9a43db9-c576-4bd4-a25d-8af7ad92303c/init-textfile/0.log"
Jul 17 22:14:49.563898 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.563872    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:14:49.568244 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.568230    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:14:49.574955 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.574860    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:14:49.688488 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.688433    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4\" (UID: \"fda113c3-e922-4558-81ce-8d9a475cca34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:14:49.788763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.788733    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4\" (UID: \"fda113c3-e922-4558-81ce-8d9a475cca34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:14:49.789240 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.789221    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4\" (UID: \"fda113c3-e922-4558-81ce-8d9a475cca34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:14:49.879845 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:49.879816    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:14:50.010151 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.010129    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:14:50.011854 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:14:50.011826    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda113c3_e922_4558_81ce_8d9a475cca34.slice/crio-7a346389106bf5cd7e23ecc2a5aeaf534ea290295ef037cb728c164d01ddc426 WatchSource:0}: Error finding container 7a346389106bf5cd7e23ecc2a5aeaf534ea290295ef037cb728c164d01ddc426: Status 404 returned error can't find the container with id 7a346389106bf5cd7e23ecc2a5aeaf534ea290295ef037cb728c164d01ddc426
Jul 17 22:14:50.230185 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.230125    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerStarted","Data":"a3fcf3984a831440db6d27fab8a9e527f085e81bc546f9358f4b10e0dce3a591"}
Jul 17 22:14:50.230185 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.230180    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerStarted","Data":"7a346389106bf5cd7e23ecc2a5aeaf534ea290295ef037cb728c164d01ddc426"}
Jul 17 22:14:50.246576 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.246552    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:50.248934 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.248899    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" podStartSLOduration=1.248884714 podStartE2EDuration="1.248884714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-17 22:14:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:50.246645047 +0000 UTC m=+3466.227015537" watchObservedRunningTime="2026-07-17 22:14:50.248884714 +0000 UTC m=+3466.229255191"
Jul 17 22:14:50.364281 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.364259    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:50.495669 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.495646    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca\") pod \"6289ab0b-4098-4b37-9889-40e91ebb7a53\" (UID: \"6289ab0b-4098-4b37-9889-40e91ebb7a53\") "
Jul 17 22:14:50.495990 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.495967    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6289ab0b-4098-4b37-9889-40e91ebb7a53" (UID: "6289ab0b-4098-4b37-9889-40e91ebb7a53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:50.597120 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:50.597091    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6289ab0b-4098-4b37-9889-40e91ebb7a53-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:51.170053 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.169982    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"]
Jul 17 22:14:51.170285 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170273    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170286    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170299    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170304    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170315    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170333 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170320    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170369    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170377    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.170505 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.170385    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" containerName="revoke-keys"
Jul 17 22:14:51.180087 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.180055    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.181986 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.181837    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"]
Jul 17 22:14:51.182387 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.182362    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjx6d\"/\"openshift-service-ca.crt\""
Jul 17 22:14:51.182533 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.182400    2546 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-gjx6d\"/\"kube-root-ca.crt\""
Jul 17 22:14:51.183083 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.183063    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-gjx6d\"/\"default-dockercfg-sgfdh\""
Jul 17 22:14:51.237128 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.237108    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"
Jul 17 22:14:51.237246 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.237146    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2" event={"ID":"6289ab0b-4098-4b37-9889-40e91ebb7a53","Type":"ContainerDied","Data":"81f06fa4f7ff5ad9f14057055e30b72c80532b54d136646d94ebc987799d87c1"}
Jul 17 22:14:51.237246 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.237206    2546 scope.go:117] "RemoveContainer" containerID="e0093ee4f18a58bd563a57df06e9c60540a0554df57b0fcfd7e831c02b98e5ae"
Jul 17 22:14:51.238956 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.238908    2546 generic.go:356] "Generic (PLEG): container finished" podID="fda113c3-e922-4558-81ce-8d9a475cca34" containerID="a3fcf3984a831440db6d27fab8a9e527f085e81bc546f9358f4b10e0dce3a591" exitCode=6
Jul 17 22:14:51.239065 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.239014    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerDied","Data":"a3fcf3984a831440db6d27fab8a9e527f085e81bc546f9358f4b10e0dce3a591"}
Jul 17 22:14:51.239168 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.239150    2546 scope.go:117] "RemoveContainer" containerID="a3fcf3984a831440db6d27fab8a9e527f085e81bc546f9358f4b10e0dce3a591"
Jul 17 22:14:51.270519 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.270495    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:51.276177 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.276156    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-ddd2a457-brlp2"]
Jul 17 22:14:51.302553 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.302531    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpgrv\" (UniqueName: \"kubernetes.io/projected/c140a1b7-1634-4a73-9841-da8eb13e2887-kube-api-access-kpgrv\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.302629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.302557    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-podres\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.302629 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.302575    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-lib-modules\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.302745 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.302695    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-proc\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.302801 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.302756    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-sys\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403653 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403629    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpgrv\" (UniqueName: \"kubernetes.io/projected/c140a1b7-1634-4a73-9841-da8eb13e2887-kube-api-access-kpgrv\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403657    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-podres\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403674    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-lib-modules\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403698    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-proc\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403756 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403716    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-sys\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403798    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-proc\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403824    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-sys\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403836    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-podres\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.403924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.403841    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c140a1b7-1634-4a73-9841-da8eb13e2887-lib-modules\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.411344 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.411322    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpgrv\" (UniqueName: \"kubernetes.io/projected/c140a1b7-1634-4a73-9841-da8eb13e2887-kube-api-access-kpgrv\") pod \"perf-node-gather-daemonset-66nbm\" (UID: \"c140a1b7-1634-4a73-9841-da8eb13e2887\") " pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.490247 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.490228    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:51.617962 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:51.617929    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"]
Jul 17 22:14:51.620607 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:14:51.620586    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podc140a1b7_1634_4a73_9841_da8eb13e2887.slice/crio-b1c38db79faa19affcdeeffb2b0b3dac023c12eabdd7fbb3bb3ab09cfcd71a7a WatchSource:0}: Error finding container b1c38db79faa19affcdeeffb2b0b3dac023c12eabdd7fbb3bb3ab09cfcd71a7a: Status 404 returned error can't find the container with id b1c38db79faa19affcdeeffb2b0b3dac023c12eabdd7fbb3bb3ab09cfcd71a7a
Jul 17 22:14:52.246588 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.246551    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm" event={"ID":"c140a1b7-1634-4a73-9841-da8eb13e2887","Type":"ContainerStarted","Data":"ed46527157aa2ae1d7bdff97cf8fe62f41b4122407ef474db995f5c426f14ebb"}
Jul 17 22:14:52.246847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.246616    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm" event={"ID":"c140a1b7-1634-4a73-9841-da8eb13e2887","Type":"ContainerStarted","Data":"b1c38db79faa19affcdeeffb2b0b3dac023c12eabdd7fbb3bb3ab09cfcd71a7a"}
Jul 17 22:14:52.246847 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.246766    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:52.249458 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.249427    2546 generic.go:356] "Generic (PLEG): container finished" podID="fda113c3-e922-4558-81ce-8d9a475cca34" containerID="41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895" exitCode=6
Jul 17 22:14:52.249600 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.249457    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerDied","Data":"41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895"}
Jul 17 22:14:52.249600 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.249513    2546 scope.go:117] "RemoveContainer" containerID="a3fcf3984a831440db6d27fab8a9e527f085e81bc546f9358f4b10e0dce3a591"
Jul 17 22:14:52.249735 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.249700    2546 scope.go:117] "RemoveContainer" containerID="41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895"
Jul 17 22:14:52.249948 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:52.249923    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4_odh-ai-gateway-infra(fda113c3-e922-4558-81ce-8d9a475cca34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" podUID="fda113c3-e922-4558-81ce-8d9a475cca34"
Jul 17 22:14:52.265076 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.265043    2546 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm" podStartSLOduration=1.265030122 podStartE2EDuration="1.265030122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-17 22:14:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-17 22:14:52.263246209 +0000 UTC m=+3468.243616687" watchObservedRunningTime="2026-07-17 22:14:52.265030122 +0000 UTC m=+3468.245400599"
Jul 17 22:14:52.535936 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:52.535874    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6289ab0b-4098-4b37-9889-40e91ebb7a53" path="/var/lib/kubelet/pods/6289ab0b-4098-4b37-9889-40e91ebb7a53/volumes"
Jul 17 22:14:53.256472 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:53.256435    2546 scope.go:117] "RemoveContainer" containerID="41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895"
Jul 17 22:14:53.256970 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:53.256663    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4_odh-ai-gateway-infra(fda113c3-e922-4558-81ce-8d9a475cca34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" podUID="fda113c3-e922-4558-81ce-8d9a475cca34"
Jul 17 22:14:54.027040 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.027016    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zsxq7_e6c349f1-c206-49d8-a9df-8ab81a3b68ea/dns/0.log"
Jul 17 22:14:54.048518 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.048496    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zsxq7_e6c349f1-c206-49d8-a9df-8ab81a3b68ea/kube-rbac-proxy/0.log"
Jul 17 22:14:54.098036 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.098011    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-dktcq_51dc316d-79fd-4a8d-9f06-e44e9e84e2e5/dns-node-resolver/0.log"
Jul 17 22:14:54.542145 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.542118    2546 scope.go:117] "RemoveContainer" containerID="90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049"
Jul 17 22:14:54.654594 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.654569    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-549479f6c5-v5cqb_b7d800b3-023c-498b-af08-9ec4308ea109/registry/0.log"
Jul 17 22:14:54.723565 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:54.723540    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-vqd2w_54dc74f1-96b7-474e-9fe5-cb1128657650/node-ca/0.log"
Jul 17 22:14:55.269310 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:55.269282    2546 generic.go:356] "Generic (PLEG): container finished" podID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerID="f36e1232731d1e7dec3834ddaf5eea01b2763448ced869ab2b126c1051c70c29" exitCode=6
Jul 17 22:14:55.269508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:55.269360    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerDied","Data":"f36e1232731d1e7dec3834ddaf5eea01b2763448ced869ab2b126c1051c70c29"}
Jul 17 22:14:55.269508 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:55.269429    2546 scope.go:117] "RemoveContainer" containerID="90e478fd48649d1cbb778b4435645412f651669f3f0eabe5a635f292023f8049"
Jul 17 22:14:55.269674 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:55.269658    2546 scope.go:117] "RemoveContainer" containerID="f36e1232731d1e7dec3834ddaf5eea01b2763448ced869ab2b126c1051c70c29"
Jul 17 22:14:55.269900 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:14:55.269869    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh_odh-ai-gateway-infra(ebd648af-15c5-4f46-99e9-3bc9f8a3db16)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16"
Jul 17 22:14:55.765930 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:55.765894    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gzv5k_ad5b0183-f8a6-4173-9c16-4a91202649b0/discovery/0.log"
Jul 17 22:14:56.296997 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.296971    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:56.421623 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.421602    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:56.544153 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.544116    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca\") pod \"ebd648af-15c5-4f46-99e9-3bc9f8a3db16\" (UID: \"ebd648af-15c5-4f46-99e9-3bc9f8a3db16\") "
Jul 17 22:14:56.544442 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.544420    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ebd648af-15c5-4f46-99e9-3bc9f8a3db16" (UID: "ebd648af-15c5-4f46-99e9-3bc9f8a3db16"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:14:56.644536 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.644475    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ebd648af-15c5-4f46-99e9-3bc9f8a3db16-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:14:56.683278 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:56.683256    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-jhp4h_a9628a26-4ddd-46b2-8a74-39731b45c70c/serve-healthcheck-canary/0.log"
Jul 17 22:14:57.270133 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.270098    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxq8w_323ca98f-d27a-455c-9dea-22cfd7e88e9e/kube-rbac-proxy/0.log"
Jul 17 22:14:57.283504 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.283480    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"
Jul 17 22:14:57.283662 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.283512    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh" event={"ID":"ebd648af-15c5-4f46-99e9-3bc9f8a3db16","Type":"ContainerDied","Data":"b90da46c0ded19ae8ab2007709059ee6ac6a7d490757c2333f77911540c1e103"}
Jul 17 22:14:57.283662 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.283594    2546 scope.go:117] "RemoveContainer" containerID="f36e1232731d1e7dec3834ddaf5eea01b2763448ced869ab2b126c1051c70c29"
Jul 17 22:14:57.292384 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.292363    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxq8w_323ca98f-d27a-455c-9dea-22cfd7e88e9e/exporter/0.log"
Jul 17 22:14:57.307476 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.307455    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:57.313907 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.313883    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-24f6a20e-mn9fh"]
Jul 17 22:14:57.316970 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:57.316955    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-rxq8w_323ca98f-d27a-455c-9dea-22cfd7e88e9e/extractor/0.log"
Jul 17 22:14:58.267514 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:58.267481    2546 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-gjx6d/perf-node-gather-daemonset-66nbm"
Jul 17 22:14:58.536428 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:14:58.536350    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" path="/var/lib/kubelet/pods/ebd648af-15c5-4f46-99e9-3bc9f8a3db16/volumes"
Jul 17 22:15:00.010763 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.010733    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6dbd9688-wf99g_789287cc-65cb-4d08-9c36-320a8146443e/manager/0.log"
Jul 17 22:15:00.046567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.046550    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6d6dbd9688-wf99g_789287cc-65cb-4d08-9c36-320a8146443e/copy-manifests/0.log"
Jul 17 22:15:00.188601 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188534    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"]
Jul 17 22:15:00.188872 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188859    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.188924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188874    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.188924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188897    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.188924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188903    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.189018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188950    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.189018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188958    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.189018 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.188963    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:00.193096 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.193080    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"
Jul 17 22:15:00.200408 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.200340    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"]
Jul 17 22:15:00.203661 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.203632    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"
Jul 17 22:15:00.334635 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:00.334608    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"]
Jul 17 22:15:00.335906 ip-10-0-135-220 kubenswrapper[2546]: W0717 22:15:00.335872    2546 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf70003_9d59_442a_89ee_116ef7a31cf2.slice/crio-d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240 WatchSource:0}: Error finding container d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240: Status 404 returned error can't find the container with id d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240
Jul 17 22:15:01.309810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:01.309781    2546 generic.go:356] "Generic (PLEG): container finished" podID="adf70003-9d59-442a-89ee-116ef7a31cf2" containerID="336c3c4e8c28fa9dc031b8c9e81fc811591d6529fa903d3d59f17f3df7b56116" exitCode=0
Jul 17 22:15:01.310127 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:01.309865    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2" event={"ID":"adf70003-9d59-442a-89ee-116ef7a31cf2","Type":"ContainerDied","Data":"336c3c4e8c28fa9dc031b8c9e81fc811591d6529fa903d3d59f17f3df7b56116"}
Jul 17 22:15:01.310127 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:01.309913    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2" event={"ID":"adf70003-9d59-442a-89ee-116ef7a31cf2","Type":"ContainerStarted","Data":"d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240"}
Jul 17 22:15:01.609007 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:01.608934    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76f7bc5b6f-qd4k2_b5f0e76e-e33a-4fa0-82d9-af195318c546/manager/1.log"
Jul 17 22:15:01.609250 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:01.609236    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-76f7bc5b6f-qd4k2_b5f0e76e-e33a-4fa0-82d9-af195318c546/manager/0.log"
Jul 17 22:15:02.387031 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:02.386995    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gzv5k_ad5b0183-f8a6-4173-9c16-4a91202649b0/discovery/0.log"
Jul 17 22:15:02.457761 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:02.457741    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"
Jul 17 22:15:03.322573 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.322534    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2" event={"ID":"adf70003-9d59-442a-89ee-116ef7a31cf2","Type":"ContainerDied","Data":"d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240"}
Jul 17 22:15:03.322573 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.322554    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29738775-wb5w2"
Jul 17 22:15:03.322871 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.322608    2546 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5f3bcfc184615888a1b8584b2d6b73db71ddf73ac935ae117a5dba7cc9f9240"
Jul 17 22:15:03.516651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.516621    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"]
Jul 17 22:15:03.520567 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.520543    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29738730-j4h9q"]
Jul 17 22:15:03.772737 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:03.772698    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-gzv5k_ad5b0183-f8a6-4173-9c16-4a91202649b0/discovery/0.log"
Jul 17 22:15:04.534193 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:04.534169    2546 scope.go:117] "RemoveContainer" containerID="41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895"
Jul 17 22:15:04.536557 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:04.536536    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="263b66e2-e7e7-4393-9269-5aad90490125" path="/var/lib/kubelet/pods/263b66e2-e7e7-4393-9269-5aad90490125/volumes"
Jul 17 22:15:05.319852 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:05.319823    2546 scope.go:117] "RemoveContainer" containerID="599d82eb3bd883f2b3db4380b35be6faf497a601170435aea0c11a600818429e"
Jul 17 22:15:05.335547 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:05.335523    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerStarted","Data":"cece029e426d995fd0cbd4759da351b522a98812564ab24dc2e91ed9a1910151"}
Jul 17 22:15:06.341849 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.341813    2546 generic.go:356] "Generic (PLEG): container finished" podID="fda113c3-e922-4558-81ce-8d9a475cca34" containerID="cece029e426d995fd0cbd4759da351b522a98812564ab24dc2e91ed9a1910151" exitCode=6
Jul 17 22:15:06.342273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.341888    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerDied","Data":"cece029e426d995fd0cbd4759da351b522a98812564ab24dc2e91ed9a1910151"}
Jul 17 22:15:06.342273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.341953    2546 scope.go:117] "RemoveContainer" containerID="41c6b2e261d7d9a98b56d29b9fd825d0454adba76d85d7354a4f699f89f1c895"
Jul 17 22:15:06.342273 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.342207    2546 scope.go:117] "RemoveContainer" containerID="cece029e426d995fd0cbd4759da351b522a98812564ab24dc2e91ed9a1910151"
Jul 17 22:15:06.342412 ip-10-0-135-220 kubenswrapper[2546]: E0717 22:15:06.342389    2546 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4_odh-ai-gateway-infra(fda113c3-e922-4558-81ce-8d9a475cca34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" podUID="fda113c3-e922-4558-81ce-8d9a475cca34"
Jul 17 22:15:06.365671 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.365647    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:15:06.542924 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.542897    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-8jv2q_f4cb5585-d50e-4c20-9573-8ba754f22652/migrator/0.log"
Jul 17 22:15:06.563551 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:06.563530    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7cc9c7ffd7-8jv2q_f4cb5585-d50e-4c20-9573-8ba754f22652/graceful-termination/0.log"
Jul 17 22:15:07.479419 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:07.479395    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:15:07.623365 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:07.623304    2546 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca\") pod \"fda113c3-e922-4558-81ce-8d9a475cca34\" (UID: \"fda113c3-e922-4558-81ce-8d9a475cca34\") "
Jul 17 22:15:07.623608 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:07.623581    2546 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fda113c3-e922-4558-81ce-8d9a475cca34" (UID: "fda113c3-e922-4558-81ce-8d9a475cca34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 17 22:15:07.724651 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:07.724625    2546 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fda113c3-e922-4558-81ce-8d9a475cca34-maas-api-service-ca\") on node \"ip-10-0-135-220.ec2.internal\" DevicePath \"\""
Jul 17 22:15:07.982696 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:07.982647    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9jpfn_fc22caa7-9281-44f6-9738-4edffb7a5237/kube-multus/0.log"
Jul 17 22:15:08.153066 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.153042    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/kube-multus-additional-cni-plugins/0.log"
Jul 17 22:15:08.173563 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.173538    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/egress-router-binary-copy/0.log"
Jul 17 22:15:08.196165 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.196146    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/cni-plugins/0.log"
Jul 17 22:15:08.218979 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.218960    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/bond-cni-plugin/0.log"
Jul 17 22:15:08.238862 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.238847    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/routeoverride-cni/0.log"
Jul 17 22:15:08.258839 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.258821    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/whereabouts-cni-bincopy/0.log"
Jul 17 22:15:08.280369 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.280352    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-9cgfc_b196a3d2-d251-432c-bcfd-abb6171d087b/whereabouts-cni/0.log"
Jul 17 22:15:08.355030 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.355011    2546 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4" event={"ID":"fda113c3-e922-4558-81ce-8d9a475cca34","Type":"ContainerDied","Data":"7a346389106bf5cd7e23ecc2a5aeaf534ea290295ef037cb728c164d01ddc426"}
Jul 17 22:15:08.355134 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.355058    2546 scope.go:117] "RemoveContainer" containerID="cece029e426d995fd0cbd4759da351b522a98812564ab24dc2e91ed9a1910151"
Jul 17 22:15:08.355134 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.355016    2546 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"
Jul 17 22:15:08.377621 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.377600    2546 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:15:08.383616 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.383582    2546 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-da1f2be7-w96p4"]
Jul 17 22:15:08.535191 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.535140    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p8c6r_2cd6a154-86ef-464e-9ca0-6aff2fca8bb0/network-metrics-daemon/0.log"
Jul 17 22:15:08.535662 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.535639    2546 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" path="/var/lib/kubelet/pods/fda113c3-e922-4558-81ce-8d9a475cca34/volumes"
Jul 17 22:15:08.550789 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.550770    2546 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-p8c6r_2cd6a154-86ef-464e-9ca0-6aff2fca8bb0/kube-rbac-proxy/0.log"
Jul 17 22:15:08.706810 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.706785    2546 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"]
Jul 17 22:15:08.707059 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707048    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707060    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707074    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707080    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707091    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707096    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707104    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adf70003-9d59-442a-89ee-116ef7a31cf2" containerName="cleanup"
Jul 17 22:15:08.707108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707109    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="adf70003-9d59-442a-89ee-116ef7a31cf2" containerName="cleanup"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707116    2546 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707120    2546 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebd648af-15c5-4f46-99e9-3bc9f8a3db16" containerName="revoke-keys"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707165    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="adf70003-9d59-442a-89ee-116ef7a31cf2" containerName="cleanup"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707173    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707179    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.707339 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.707185    2546 memory_manager.go:368] "RemoveStaleState removing state" podUID="fda113c3-e922-4558-81ce-8d9a475cca34" containerName="revoke-keys"
Jul 17 22:15:08.711108 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.711094    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"
Jul 17 22:15:08.713308 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.713293    2546 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-bpwpc\""
Jul 17 22:15:08.717062 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.717042    2546 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"]
Jul 17 22:15:08.832879 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.832814    2546 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988291cd-a9d0-4cd9-8d10-92b2c9c1219b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk\" (UID: \"988291cd-a9d0-4cd9-8d10-92b2c9c1219b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"
Jul 17 22:15:08.933599 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.933577    2546 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988291cd-a9d0-4cd9-8d10-92b2c9c1219b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk\" (UID: \"988291cd-a9d0-4cd9-8d10-92b2c9c1219b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"
Jul 17 22:15:08.934125 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:08.934109    2546 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/988291cd-a9d0-4cd9-8d10-92b2c9c1219b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk\" (UID: \"988291cd-a9d0-4cd9-8d10-92b2c9c1219b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"
Jul 17 22:15:09.021739 ip-10-0-135-220 kubenswrapper[2546]: I0717 22:15:09.021694    2546 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b3d03bc3-dvgnk"