Jul 16 21:44:43.896839 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:44:43.896849 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:44:43.896859 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:44:43.897170 ip-10-0-138-223 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:44:54.146238 ip-10-0-138-223 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:44:54.146259 ip-10-0-138-223 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot fedc71a0b9b34225b2344115930bcae1 --
Jul 16 21:46:50.832966 ip-10-0-138-223 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:46:51.189616 ip-10-0-138-223 kubenswrapper[2581]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:51.189616 ip-10-0-138-223 kubenswrapper[2581]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:46:51.189616 ip-10-0-138-223 kubenswrapper[2581]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:51.189616 ip-10-0-138-223 kubenswrapper[2581]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:46:51.189616 ip-10-0-138-223 kubenswrapper[2581]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:46:51.191528 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.191437    2581 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:46:51.195033 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195019    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.195033 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195034    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195038    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195041    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195044    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195047    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195050    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195053    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195055    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195058    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195061    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195064    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195067    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195070    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195072    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195075    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195077    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195080    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195084    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195087    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.195112 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195090    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195093    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195096    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195099    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195101    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195103    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195106    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195109    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195111    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195114    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195117    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195119    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195122    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195124    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195127    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195130    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195132    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195135    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195138    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195140    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.195559 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195143    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195146    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195149    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195151    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195153    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195156    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195158    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195160    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195163    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195165    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195168    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195170    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195172    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195175    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195178    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195181    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195183    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195186    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195189    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195191    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.196053 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195194    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195197    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195201    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195205    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195208    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195211    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195213    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195216    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195218    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195220    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195223    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195225    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195228    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195230    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195233    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195236    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195238    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195241    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195243    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.196530 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195245    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195248    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195250    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195252    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195255    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195258    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195262    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195265    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195269    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195271    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195274    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195277    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195280    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195283    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195286    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195288    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195291    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195294    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195297    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.197039 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195299    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195986    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195993    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195996    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.195998    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196001    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196003    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196023    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196027    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196030    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196034    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196037    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196040    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196043    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196045    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196048    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196051    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196054    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196056    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196060    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.197483 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196063    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196067    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196070    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196073    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196076    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196082    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196086    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196089    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196092    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196095    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196097    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196100    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196102    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196105    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196109    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196111    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196114    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196117    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196120    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.198001 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196122    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196125    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196128    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196130    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196133    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196136    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196138    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196141    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196143    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196145    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196148    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196150    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196153    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196155    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196158    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196160    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196163    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196165    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196168    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196171    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.198456 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196173    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196175    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196178    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196180    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196183    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196185    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196187    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196190    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196192    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196194    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196197    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196199    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196201    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196204    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196206    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196208    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196214    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196217    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196220    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.198960 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196223    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196225    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196228    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196231    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196233    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196236    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196238    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196241    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196243    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196246    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196248    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196251    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196253    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196256    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196258    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196261    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196263    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196266    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196268    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196271    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.199447 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196273    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196275    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196623    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196627    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196632    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196636    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196639    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196641    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196644    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196647    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196654    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196657    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196660    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196662    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196664    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196668    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196672    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196691    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196694    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.200034 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196697    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196699    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196702    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196705    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196708    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196710    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196713    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196715    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196718    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196720    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196723    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196725    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196727    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196730    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196732    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196734    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196737    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196739    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196742    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196744    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.208542 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196747    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196749    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196751    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196754    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196757    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196760    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196762    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196765    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196767    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196769    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196773    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196777    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196779    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196782    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196785    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196788    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196791    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196793    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196796    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.209152 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196798    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196801    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196803    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196805    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196808    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196810    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196813    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196816    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196819    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196821    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196824    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196826    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196828    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196831    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196833    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196836    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196838    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196840    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196843    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196845    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.209615 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196848    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196850    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196852    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196855    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196857    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196860    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196862    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196865    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196868    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196870    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196872    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196875    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196877    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196880    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196882    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196885    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196887    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196890    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196892    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196894    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.210162 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196897    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196899    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.196902    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.196976    2581 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.196985    2581 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.196994    2581 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197000    2581 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197007    2581 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197011    2581 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197015    2581 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197019    2581 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197022    2581 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197025    2581 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197029    2581 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197032    2581 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197035    2581 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197038    2581 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197041    2581 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197044    2581 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197047    2581 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197050    2581 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197054    2581 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197057    2581 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197060    2581 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:46:51.210627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197063    2581 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197066    2581 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197069    2581 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197072    2581 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197075    2581 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197078    2581 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197081    2581 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197084    2581 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197087    2581 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197090    2581 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197092    2581 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197096    2581 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197099    2581 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197102    2581 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197104    2581 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197107    2581 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197110    2581 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197113    2581 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197116    2581 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197118    2581 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197122    2581 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197125    2581 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197128    2581 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197133    2581 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197135    2581 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:46:51.211271 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197138    2581 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197141    2581 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197144    2581 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197146    2581 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197149    2581 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197152    2581 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197154    2581 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197157    2581 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197166    2581 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197168    2581 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197171    2581 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197174    2581 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197177    2581 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197180    2581 flags.go:64] FLAG: --help="false"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197183    2581 flags.go:64] FLAG: --hostname-override="ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197186    2581 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197188    2581 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197191    2581 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197195    2581 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197198    2581 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197200    2581 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197203    2581 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197206    2581 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:46:51.211988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197208    2581 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197211    2581 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197214    2581 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197218    2581 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197221    2581 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197224    2581 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197227    2581 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197230    2581 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197233    2581 flags.go:64] FLAG: --lock-file=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197236    2581 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197238    2581 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197241    2581 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197246    2581 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197249    2581 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197252    2581 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197254    2581 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197257    2581 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197260    2581 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197263    2581 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197265    2581 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197269    2581 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197272    2581 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197276    2581 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197278    2581 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197281    2581 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:46:51.212577 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197284    2581 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197286    2581 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197289    2581 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197291    2581 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197294    2581 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197301    2581 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197303    2581 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197307    2581 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197309    2581 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197312    2581 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197319    2581 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197321    2581 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197324    2581 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197327    2581 flags.go:64] FLAG: --port="10250"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197330    2581 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197334    2581 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-04882a3d2ee0156f6"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197337    2581 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197340    2581 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197343    2581 flags.go:64] FLAG: --register-node="true"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197345    2581 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197349    2581 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197351    2581 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197354    2581 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197356    2581 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197360    2581 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:46:51.213185 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197363    2581 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197365    2581 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197368    2581 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197371    2581 flags.go:64] FLAG: --runonce="false"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197373    2581 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197376    2581 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197379    2581 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197381    2581 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197384    2581 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197387    2581 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197390    2581 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197393    2581 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197395    2581 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197398    2581 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197400    2581 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197403    2581 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197406    2581 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197410    2581 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197412    2581 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197417    2581 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197420    2581 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197423    2581 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197428    2581 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197431    2581 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197434    2581 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:46:51.213925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197436    2581 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197439    2581 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197442    2581 flags.go:64] FLAG: --v="2"
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197445    2581 flags.go:64] FLAG: --version="false"
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197449    2581 flags.go:64] FLAG: --vmodule=""
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197452    2581 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.197455    2581 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197580    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197585    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197588    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197591    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197594    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197597    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197600    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197602    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197605    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197608    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197611    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197613    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197616    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197618    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.214519 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197621    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197623    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197625    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197629    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197631    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197634    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197636    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197639    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197643    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197645    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197648    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197650    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197653    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197655    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197658    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197660    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197662    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197665    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197667    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197670    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.215066 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197672    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197689    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197693    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197697    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197700    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197705    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197707    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197711    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197714    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197718    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197721    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197723    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197726    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197728    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197731    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197734    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197737    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197739    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197742    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197744    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.215590 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197748    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197751    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197754    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197756    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197767    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197770    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197772    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197775    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197778    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197780    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197783    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197785    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197788    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197790    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197793    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197795    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197798    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197800    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197804    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.216116 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197807    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197810    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197813    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197815    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197818    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197821    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197823    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197826    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197830    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197832    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197834    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197837    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197839    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197843    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197846    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197849    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197852    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197854    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197857    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197859    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.216712 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197862    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197865    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197867    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197869    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197872    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.197874    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.198527    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.211580    2581 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.211596    2581 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.212262    2581 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.212269    2581 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212327    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212332    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212335    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.217200 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212338    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212341    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212343    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212346    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212349    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212351    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212354    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212356    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212359    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212361    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212364    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212366    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212369    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212371    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212374    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212377    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212380    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212383    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212385    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212388    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.217593 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212390    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212392    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212395    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212398    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212401    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212403    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212406    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212409    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212411    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212414    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212416    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212419    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212422    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212424    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212427    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212429    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212432    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212434    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212436    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.218109 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212439    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212441    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212444    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212446    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212449    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212452    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212454    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212458    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212463    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212466    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212469    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212472    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212474    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212477    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212480    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212483    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212485    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212488    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212491    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212493    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.218655 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212496    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212499    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212502    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212504    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212507    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212510    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212513    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212517    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212519    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212522    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212525    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212529    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212532    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212535    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212537    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212540    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212543    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212545    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212548    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.219428 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212550    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212553    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212555    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212558    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212560    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212563    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212565    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212568    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212570    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212573    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212575    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212578    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212581    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212583    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212586    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212589    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212591    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.220163 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212594    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.212599    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212744    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212750    2581 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212753    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212756    2581 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212759    2581 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212762    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212765    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212768    2581 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212770    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212773    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212776    2581 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212778    2581 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212782    2581 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:46:51.220597 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212784    2581 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212787    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212789    2581 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212792    2581 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212794    2581 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212797    2581 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212799    2581 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212802    2581 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212804    2581 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212808    2581 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212812    2581 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212815    2581 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212817    2581 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212820    2581 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212822    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212825    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212827    2581 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212830    2581 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212832    2581 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:46:51.221247 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212835    2581 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212837    2581 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212840    2581 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212843    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212845    2581 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212848    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212850    2581 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212852    2581 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212855    2581 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212857    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212860    2581 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212862    2581 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212864    2581 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212867    2581 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212872    2581 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212875    2581 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212878    2581 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212881    2581 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212884    2581 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:46:51.221775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212886    2581 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212889    2581 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212892    2581 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212894    2581 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212897    2581 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212899    2581 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212902    2581 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212904    2581 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212906    2581 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212908    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212911    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212914    2581 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212916    2581 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212918    2581 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212921    2581 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212923    2581 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212925    2581 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212928    2581 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212930    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212933    2581 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:46:51.222266 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212935    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212937    2581 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212940    2581 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212942    2581 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212945    2581 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212947    2581 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212950    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212954    2581 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212956    2581 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212959    2581 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212961    2581 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212964    2581 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212966    2581 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212969    2581 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212971    2581 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212973    2581 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212976    2581 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212978    2581 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212980    2581 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:46:51.222858 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212984    2581 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212987    2581 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212989    2581 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212992    2581 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212995    2581 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.212997    2581 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.213000    2581 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.213003    2581 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:51.213005    2581 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.213010    2581 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.213134    2581 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.215063    2581 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.216198    2581 server.go:1019] "Starting client certificate rotation"
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.216308    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:46:51.223334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.216362    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:46:51.223668 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.219057    2581 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:46:51.223668 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.221885    2581 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:46:51.232786 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.232767    2581 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:46:51.234971 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.234957    2581 log.go:25] "Validated CRI v1 image API"
Jul 16 21:46:51.235814 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.235800    2581 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:46:51.238142 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.238122    2581 fs.go:135] Filesystem UUIDs: map[3a6c3a92-50ff-47a4-8fef-5f122c0cbc1a:/dev/nvme0n1p4 722ed64a-98fa-4b7b-9814-52abfd7ea3c1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 16 21:46:51.238184 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.238148    2581 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:46:51.244648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.244527    2581 manager.go:216] Machine: {Timestamp:2026-07-16 21:46:51.243136597 +0000 UTC m=+0.306126609 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3091980 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27dc08278553888eae10bd18b228c1 SystemUUID:ec27dc08-2785-5388-8eae-10bd18b228c1 BootID:fedc71a0-b9b3-4225-b234-4115930bcae1 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:bb:ea:d9:58:a1 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:bb:ea:d9:58:a1 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:29:75:0b:1e:82 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:46:51.244648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.244639    2581 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:46:51.244818 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.244756    2581 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:46:51.246409 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.246392    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:46:51.247332 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.247308    2581 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:46:51.247470 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.247335    2581 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-223.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:46:51.248465 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.248454    2581 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:46:51.248498 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.248468    2581 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:46:51.248498 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.248475    2581 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:46:51.249165 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.249153    2581 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:46:51.250247 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.250237    2581 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:46:51.252761 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252750    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:46:51.252906 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252897    2581 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:46:51.252940 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252927    2581 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:46:51.252940 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252939    2581 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:46:51.252991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252950    2581 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:46:51.252991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252958    2581 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:46:51.252991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.252966    2581 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:46:51.253391 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.253378    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:46:51.253428 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.253404    2581 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:46:51.255453 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.255436    2581 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:46:51.257108 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.257095    2581 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:46:51.257149 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.257125    2581 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:46:51.258030 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258019    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:46:51.258063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258042    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:46:51.258063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258049    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:46:51.258063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258054    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:46:51.258063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258062    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258067    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258075    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258080    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258087    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258092    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258101    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:46:51.258162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258106    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:46:51.258867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258857    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:46:51.258867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.258867    2581 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:46:51.262017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.261821    2581 server.go:1268] "Started kubelet"
Jul 16 21:46:51.262017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.261911    2581 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:46:51.262156 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.262015    2581 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:46:51.262156 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.262083    2581 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:46:51.263448 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.263427    2581 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:46:51.263783 ip-10-0-138-223 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:46:51.265810 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.264749    2581 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-223.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:46:51.265810 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.264934    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:46:51.265810 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.265018    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:46:51.265810 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.265380    2581 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:46:51.265943 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.265822    2581 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:46:51.267988 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.267970    2581 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:46:51.268074 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.267978    2581 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:46:51.268153 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.268129    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.268512 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268494    2581 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:46:51.268512 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268500    2581 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:46:51.268671 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268531    2581 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:46:51.268671 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268645    2581 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:46:51.268671 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268653    2581 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:46:51.268857 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268844    2581 factory.go:55] Registering systemd factory
Jul 16 21:46:51.269013 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.268863    2581 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:46:51.269598 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.269579    2581 factory.go:153] Registering CRI-O factory
Jul 16 21:46:51.269663 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.269602    2581 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:46:51.270300 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.270280    2581 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:46:51.270408 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.270312    2581 factory.go:103] Registering Raw factory
Jul 16 21:46:51.270408 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.270328    2581 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:46:51.270776 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.270759    2581 manager.go:349] Starting recovery of all containers
Jul 16 21:46:51.272692 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.272646    2581 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-223.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:46:51.272833 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.272808    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:46:51.274504 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.272303    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-223.ec2.internal.18c2e361b9dde29c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-223.ec2.internal,UID:ip-10-0-138-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-223.ec2.internal,},FirstTimestamp:2026-07-16 21:46:51.26179702 +0000 UTC m=+0.324787032,LastTimestamp:2026-07-16 21:46:51.26179702 +0000 UTC m=+0.324787032,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-223.ec2.internal,}"
Jul 16 21:46:51.280666 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.280646    2581 manager.go:354] Recovery completed
Jul 16 21:46:51.281508 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.281479    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4nc4q"
Jul 16 21:46:51.285899 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.285887    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.287783 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.287766    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.287837 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.287802    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.287837 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.287817    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.288291 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.288277    2581 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:46:51.288358 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.288293    2581 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:46:51.288358 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.288310    2581 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:46:51.288358 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.288281    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.289376 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289363    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.289415 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289391    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.289415 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289410    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.289478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289441    2581 policy_none.go:49] "None policy: Start"
Jul 16 21:46:51.289478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289456    2581 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:46:51.289478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.289469    2581 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:46:51.290034 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.290017    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-4nc4q"
Jul 16 21:46:51.290178 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.289954    2581 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-223.ec2.internal.18c2e361bb6a6e35  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-223.ec2.internal,UID:ip-10-0-138-223.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-223.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-223.ec2.internal,},FirstTimestamp:2026-07-16 21:46:51.287785013 +0000 UTC m=+0.350775033,LastTimestamp:2026-07-16 21:46:51.287785013 +0000 UTC m=+0.350775033,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-223.ec2.internal,}"
Jul 16 21:46:51.290784 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.290769    2581 policy_none.go:47] "Start"
Jul 16 21:46:51.331413 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331269    2581 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:46:51.331507 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.331434    2581 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:46:51.331507 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331447    2581 server.go:87] "Starting device plugin registration server"
Jul 16 21:46:51.331645 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331634    2581 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:46:51.331708 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331645    2581 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:46:51.331785 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331768    2581 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:46:51.331858 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331852    2581 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:46:51.331915 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.331862    2581 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:46:51.332237 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.332216    2581 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:46:51.332315 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.332268    2581 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.419148 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.419113    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:46:51.420829 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.420449    2581 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:46:51.420829 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.420483    2581 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:46:51.420829 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.420502    2581 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:46:51.420829 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.420536    2581 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:46:51.423290 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.423270    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:51.432443 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.432429    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.433182 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.433168    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.433264 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.433201    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.433264 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.433218    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.433264 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.433247    2581 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.442484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.442441    2581 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.442484 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.442463    2581 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-223.ec2.internal\": node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.462002 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.461979    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.521369 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.521329    2581 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"]
Jul 16 21:46:51.521454 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.521412    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.523294 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.523280    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.523377 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.523312    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.523377 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.523328    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.525344 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.525330    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.525466 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.525440    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.525466 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.525466    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.526053 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526027    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.526113 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526059    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.526113 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526070    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.526113 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526030    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.526206 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526127    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.526206 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.526141    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.527947 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.527932    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.528004 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.527959    2581 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:46:51.529004 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.528985    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:46:51.529064 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.529012    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:46:51.529064 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.529025    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:46:51.546306 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.546282    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.550106 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.550091    2581 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-223.ec2.internal\" not found" node="ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.562091 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.562071    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.662352 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.662332    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.670284 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.670243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.670336 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.670289    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.670372 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.670313    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec3998bf518cc797a77d2a84d9ceb141-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"ec3998bf518cc797a77d2a84d9ceb141\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.763422 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.763374    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.770702 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.770752 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770725    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec3998bf518cc797a77d2a84d9ceb141-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"ec3998bf518cc797a77d2a84d9ceb141\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.770752 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770746    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.770811 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770778    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.770811 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/ec3998bf518cc797a77d2a84d9ceb141-config\") pod \"kube-apiserver-proxy-ip-10-0-138-223.ec2.internal\" (UID: \"ec3998bf518cc797a77d2a84d9ceb141\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.770811 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.770780    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9234415a6de9d755ed5f51d746845080-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal\" (UID: \"9234415a6de9d755ed5f51d746845080\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.848861 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.848830    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.853143 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:51.853020    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:51.864189 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.864170    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:51.964738 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:51.964714    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.065238 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:52.065173    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.164200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.164179    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:52.165809 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:52.165796    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.215861 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.215845    2581 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:46:52.216239 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.215966    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:46:52.216239 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.216017    2581 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:46:52.258145 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.258119    2581 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:52.266222 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:52.266205    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.293468 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.293444    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:41:53 +0000 UTC" deadline="2027-12-31 18:50:29.212468528 +0000 UTC"
Jul 16 21:46:52.293468 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.293464    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12789h3m36.919006876s"
Jul 16 21:46:52.366776 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:52.366714    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.467648 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:52.467618    2581 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-223.ec2.internal\" not found"
Jul 16 21:46:52.513060 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:52.513030    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9234415a6de9d755ed5f51d746845080.slice/crio-0240a2f0b4f2e155387aab2e82b298dbb7adf5b96aa8ab31fcd6da0c3d53fde5 WatchSource:0}: Error finding container 0240a2f0b4f2e155387aab2e82b298dbb7adf5b96aa8ab31fcd6da0c3d53fde5: Status 404 returned error can't find the container with id 0240a2f0b4f2e155387aab2e82b298dbb7adf5b96aa8ab31fcd6da0c3d53fde5
Jul 16 21:46:52.516887 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.516872    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:46:52.527526 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.527509    2581 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:52.568352 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.568323    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:52.579389 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.579370    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:46:52.580472 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.580460    2581 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal"
Jul 16 21:46:52.589035 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:52.589020    2581 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:46:54.745264 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:54.745194    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec3998bf518cc797a77d2a84d9ceb141.slice/crio-6c8049e63044cfc8d1a3fd752ea2bac45d967eb7065f75a0ad4b44d36fda4e11 WatchSource:0}: Error finding container 6c8049e63044cfc8d1a3fd752ea2bac45d967eb7065f75a0ad4b44d36fda4e11: Status 404 returned error can't find the container with id 6c8049e63044cfc8d1a3fd752ea2bac45d967eb7065f75a0ad4b44d36fda4e11
Jul 16 21:46:54.872007 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.871936    2581 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:54.987134 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.987107    2581 apiserver.go:52] "Watching apiserver"
Jul 16 21:46:54.994028 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.994000    2581 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:46:54.994446 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.994426    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-target-qtz7s","kube-system/global-pull-secret-syncer-zqkd4","kube-system/konnectivity-agent-ss5km","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8","openshift-dns/node-resolver-gq6wg","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal","openshift-network-operator/iptables-alerter-xsfvt","openshift-ovn-kubernetes/ovnkube-node-79rh6","kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal","openshift-cluster-node-tuning-operator/tuned-kqtnz","openshift-image-registry/node-ca-q7lj4","openshift-multus/multus-additional-cni-plugins-vvs4k","openshift-multus/multus-vr858","openshift-multus/network-metrics-daemon-sxzbc"]
Jul 16 21:46:54.997039 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.997019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:54.999445 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:54.999422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.001482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.001465    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.002116 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.002097    2581 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:46:55.005250 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005070    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:46:55.005250 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005207    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:46:55.005454 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.005454 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005367    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-cvdww\""
Jul 16 21:46:55.005454 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005427    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.005628 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005545    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.005697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005668    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:46:55.005697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005688    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.005804 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005740    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-8rvzk\""
Jul 16 21:46:55.005804 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005792    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.005911 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005822    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:46:55.005911 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-pbw5s\""
Jul 16 21:46:55.005911 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.005858    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:46:55.007273 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.007259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.009537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.009516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.009617 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.009582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:46:55.010045 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010000    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:46:55.010143 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.011015 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010891    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-host-slash\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.011015 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010931    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-socket-dir-parent\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011015 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-multus\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011015 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.010982    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-hostroot\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011057    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011091    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cnibin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-netns\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011166    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-sys-fs\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011202    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5dw\" (UniqueName: \"kubernetes.io/projected/3432cb8b-9202-4e9a-a1de-d26494fa415b-kube-api-access-rx5dw\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011253    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjvmp\" (UniqueName: \"kubernetes.io/projected/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-kube-api-access-jjvmp\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011279    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-daemon-config\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-registration-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-device-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-kubelet\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-conf-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-etc-kubernetes\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011472    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-k8s-cni-cncf-io\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011496    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tgfn\" (UniqueName: \"kubernetes.io/projected/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-kube-api-access-6tgfn\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-iptables-alerter-script\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-system-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.011570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011565    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cni-binary-copy\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-bin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011626    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-multus-certs\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011650    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/383e7ef0-501b-416f-ba30-712db4ed855b-konnectivity-ca\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011692    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-socket-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011717    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-os-release\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011741    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/383e7ef0-501b-416f-ba30-712db4ed855b-agent-certs\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.011745    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-knd79\""
Jul 16 21:46:55.012119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.012000    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.012437 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.012417    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.013797 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.013775    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.014396 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.014381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:55.014481 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.014437    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:46:55.016987 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.016968    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.017086 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.017028    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:46:55.019342 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.019320    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.019478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.019354    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:46:55.019478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.019457    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:46:55.019594 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.019563    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:46:55.019594 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.019582    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.020370 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.020029    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.020370 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.020069    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.020370 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.020085    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r9k2x\""
Jul 16 21:46:55.020370 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.020224    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-kxj2v\""
Jul 16 21:46:55.020653 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.020371    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:46:55.022422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.022403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.023648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.023632    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.023879 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.023860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.023964 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.023917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-cbvld\""
Jul 16 21:46:55.024821 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.024789    2581 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:46:55.025691 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.025672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.025794 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.025746    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:46:55.025856 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.025817    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:46:55.025856 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.025844    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-n866j\""
Jul 16 21:46:55.025957 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.025849    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:46:55.034252 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.034215    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:46:55.034428 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.034217    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:46:55.034593 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.034512    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-l4phh\""
Jul 16 21:46:55.070077 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.070055    2581 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-q49ls"
Jul 16 21:46:55.088464 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.088446    2581 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-q49ls"
Jul 16 21:46:55.103465 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.103442    2581 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:46:55.111880 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111859    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-etc-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.111984 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111899    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjvmp\" (UniqueName: \"kubernetes.io/projected/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-kube-api-access-jjvmp\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.111984 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111928    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:55.111984 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-dbus\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.111984 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111974    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-systemd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.111995    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-config\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112043    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-device-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-conf-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112117    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-device-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112121    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-etc-kubernetes\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.112186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-conf\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112205    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-etc-kubernetes\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112207    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-tmp\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112256    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-k8s-cni-cncf-io\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112269    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112282    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tgfn\" (UniqueName: \"kubernetes.io/projected/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-kube-api-access-6tgfn\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112296    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-conf-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112326    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-k8s-cni-cncf-io\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112346    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112362    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-sys\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112500    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-iptables-alerter-script\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112517    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cni-binary-copy\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112548    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-bin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.112585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112589    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-bin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112589    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-multus-certs\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112627    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-multus-certs\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112663    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/383e7ef0-501b-416f-ba30-712db4ed855b-konnectivity-ca\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112698    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-ovn\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-kubernetes\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-socket-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/383e7ef0-501b-416f-ba30-712db4ed855b-agent-certs\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112801    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-log-socket\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112823    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysconfig\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112886    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-socket-dir-parent\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112935    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-slash\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112960    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.112984    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-bin\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113018    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-run\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113042    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-iptables-alerter-script\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.113084 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-cnibin\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113116    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cni-binary-copy\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-hostroot\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113171    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7thjw\" (UniqueName: \"kubernetes.io/projected/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-kube-api-access-7thjw\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113195    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113243    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-systemd\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113267    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-tuned\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113349    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-socket-dir-parent\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113378    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-hostroot\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113426    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113424    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-socket-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-node-log\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113498    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-script-lib\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113525    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-sys-fs\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5dw\" (UniqueName: \"kubernetes.io/projected/3432cb8b-9202-4e9a-a1de-d26494fa415b-kube-api-access-rx5dw\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113581    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-sys-fs\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-daemon-config\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.113808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113649    2581 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113702    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a599a220-6fa7-4d9e-be51-52258ab5f6c0-serviceca\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/383e7ef0-501b-416f-ba30-712db4ed855b-konnectivity-ca\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113872    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113921    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.113993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114022    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-hosts-file\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114046    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-registration-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114083    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114090    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-registration-dir\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-kubelet\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114123    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-daemon-config\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114128    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-kubelet-config\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114154    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-lib-modules\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v9nq\" (UniqueName: \"kubernetes.io/projected/a599a220-6fa7-4d9e-be51-52258ab5f6c0-kube-api-access-7v9nq\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114177    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-kubelet\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114183    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.114422 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114199    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-multus-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114266    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-netd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v58xx\" (UniqueName: \"kubernetes.io/projected/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-kube-api-access-v58xx\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114327    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-system-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114369    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-kubelet\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114393    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-systemd-units\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114418    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-env-overrides\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114444    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-os-release\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-system-cni-dir\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-os-release\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftgcp\" (UniqueName: \"kubernetes.io/projected/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-kube-api-access-ftgcp\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114529    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql9zs\" (UniqueName: \"kubernetes.io/projected/a8713e00-9c4c-419a-bc37-20954386ea69-kube-api-access-ql9zs\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114554    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-host-slash\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-netns\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114579    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-os-release\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114598    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-var-lib-kubelet\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.115088 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-host\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114626    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-host-slash\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114640    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a599a220-6fa7-4d9e-be51-52258ab5f6c0-host\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114670    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-multus\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114694    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-modprobe-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-tmp-dir\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114743    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97jb9\" (UniqueName: \"kubernetes.io/projected/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-kube-api-access-97jb9\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-var-lib-cni-multus\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114772    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114864    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cnibin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114934    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-cnibin\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-netns\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.115001    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-var-lib-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.114892    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/3432cb8b-9202-4e9a-a1de-d26494fa415b-etc-selinux\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.115648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.115061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-host-run-netns\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.117450 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.117427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/383e7ef0-501b-416f-ba30-712db4ed855b-agent-certs\") pod \"konnectivity-agent-ss5km\" (UID: \"383e7ef0-501b-416f-ba30-712db4ed855b\") " pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.137025 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.136888    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tgfn\" (UniqueName: \"kubernetes.io/projected/fff8bdec-cd9f-413d-8d0a-9b5fdb06b759-kube-api-access-6tgfn\") pod \"multus-vr858\" (UID: \"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759\") " pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.138824 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.138799    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5dw\" (UniqueName: \"kubernetes.io/projected/3432cb8b-9202-4e9a-a1de-d26494fa415b-kube-api-access-rx5dw\") pod \"aws-ebs-csi-driver-node-86qp8\" (UID: \"3432cb8b-9202-4e9a-a1de-d26494fa415b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.139368 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.139351    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjvmp\" (UniqueName: \"kubernetes.io/projected/1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc-kube-api-access-jjvmp\") pod \"iptables-alerter-xsfvt\" (UID: \"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc\") " pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.159979 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.159930    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"ec3998bf518cc797a77d2a84d9ceb141","Type":"ContainerStarted","Data":"6c8049e63044cfc8d1a3fd752ea2bac45d967eb7065f75a0ad4b44d36fda4e11"}
Jul 16 21:46:55.161238 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.161207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9234415a6de9d755ed5f51d746845080","Type":"ContainerStarted","Data":"0240a2f0b4f2e155387aab2e82b298dbb7adf5b96aa8ab31fcd6da0c3d53fde5"}
Jul 16 21:46:55.215342 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-node-log\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.215342 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215345    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-script-lib\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a599a220-6fa7-4d9e-be51-52258ab5f6c0-serviceca\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215421    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-node-log\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.215522 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215512    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-hosts-file\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-kubelet-config\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215561    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-lib-modules\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v9nq\" (UniqueName: \"kubernetes.io/projected/a599a220-6fa7-4d9e-be51-52258ab5f6c0-kube-api-access-7v9nq\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215629    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-netd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v58xx\" (UniqueName: \"kubernetes.io/projected/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-kube-api-access-v58xx\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.215716 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215701    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-kubelet\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-systemd-units\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215751    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-env-overrides\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215773    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-os-release\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215797    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftgcp\" (UniqueName: \"kubernetes.io/projected/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-kube-api-access-ftgcp\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215821    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql9zs\" (UniqueName: \"kubernetes.io/projected/a8713e00-9c4c-419a-bc37-20954386ea69-kube-api-access-ql9zs\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-netns\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-var-lib-kubelet\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215893    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-host\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215910    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a599a220-6fa7-4d9e-be51-52258ab5f6c0-host\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215931    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-modprobe-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216029 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.215952    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-tmp-dir\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216032    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97jb9\" (UniqueName: \"kubernetes.io/projected/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-kube-api-access-97jb9\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216095    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-var-lib-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-etc-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216144    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216168    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-dbus\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216193    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-systemd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216219    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-config\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-conf\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-tmp\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216347    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216418 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216394    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-sys\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-ovn\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-kubernetes\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216483    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-log-socket\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysconfig\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216550    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216566    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-script-lib\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216575    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-slash\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-bin\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-run\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216673    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-cnibin\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7thjw\" (UniqueName: \"kubernetes.io/projected/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-kube-api-access-7thjw\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216728    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216754    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-systemd\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216792    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-tuned\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.216884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216820    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216908    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-system-cni-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216933    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a599a220-6fa7-4d9e-be51-52258ab5f6c0-serviceca\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.216963    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-hosts-file\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217007    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-kubelet-config\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-lib-modules\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217338    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-binary-copy\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217455    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-tuning-conf-dir\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217493    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-systemd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.217579 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.217561    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:55.217876 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.217634    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:55.717598829 +0000 UTC m=+3.047038710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:55.217876 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217811    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.217876 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.217856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-netd\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218104 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218080    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-kubelet\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218176 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218110    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovnkube-config\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218176 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218133    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-systemd-units\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218306 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.218397 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218372    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysctl-conf\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.218503 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218468    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-env-overrides\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218551 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218535    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-os-release\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.218721 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218700    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-slash\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218810 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.218795    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:55.218868 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.218853    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:46:55.718839498 +0000 UTC m=+3.048279380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:55.218929 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218920    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-netns\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.218987 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.218971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-var-lib-kubelet\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219064 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219023    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-host\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219119 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a599a220-6fa7-4d9e-be51-52258ab5f6c0-host\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.219175 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219148    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-modprobe-d\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219296 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219272    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a8713e00-9c4c-419a-bc37-20954386ea69-cnibin\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.219371 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-kubernetes\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219371 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-sys\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219371 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-run-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-run-ovn\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219417    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-run\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-tmp-dir\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219441    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-cni-bin\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219477    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-var-lib-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219537 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219497    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-etc-openvswitch\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219775 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219548    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-systemd\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219775 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-sysconfig\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.219775 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-log-socket\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.219775 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219738    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-dbus\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.219898 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.219849    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.220242 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.220201    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a8713e00-9c4c-419a-bc37-20954386ea69-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.229376 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.224651    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-etc-tuned\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.229376 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.224957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-ovn-node-metrics-cert\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.230324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.230247    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-tmp\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.232296 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.232137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v9nq\" (UniqueName: \"kubernetes.io/projected/a599a220-6fa7-4d9e-be51-52258ab5f6c0-kube-api-access-7v9nq\") pod \"node-ca-q7lj4\" (UID: \"a599a220-6fa7-4d9e-be51-52258ab5f6c0\") " pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.236964 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.236939    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:46:55.237056 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.236967    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:46:55.237056 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.236982    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:55.237056 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.237034    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:55.737018606 +0000 UTC m=+3.066458490 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:55.239874 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.239848    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql9zs\" (UniqueName: \"kubernetes.io/projected/a8713e00-9c4c-419a-bc37-20954386ea69-kube-api-access-ql9zs\") pod \"multus-additional-cni-plugins-vvs4k\" (UID: \"a8713e00-9c4c-419a-bc37-20954386ea69\") " pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.240677 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.240656    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7thjw\" (UniqueName: \"kubernetes.io/projected/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-kube-api-access-7thjw\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.241984 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.241960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftgcp\" (UniqueName: \"kubernetes.io/projected/7befa2ae-6b26-46a9-a769-de9e01d7c4b3-kube-api-access-ftgcp\") pod \"ovnkube-node-79rh6\" (UID: \"7befa2ae-6b26-46a9-a769-de9e01d7c4b3\") " pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.242729 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.242704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v58xx\" (UniqueName: \"kubernetes.io/projected/cb68d5ac-5e6e-434e-9bff-7c296bff4a90-kube-api-access-v58xx\") pod \"tuned-kqtnz\" (UID: \"cb68d5ac-5e6e-434e-9bff-7c296bff4a90\") " pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.243146 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.243107    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97jb9\" (UniqueName: \"kubernetes.io/projected/0cc7a5d8-eda1-4e1d-9421-a8933d99c709-kube-api-access-97jb9\") pod \"node-resolver-gq6wg\" (UID: \"0cc7a5d8-eda1-4e1d-9421-a8933d99c709\") " pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.309215 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.309191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-xsfvt"
Jul 16 21:46:55.316835 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.316807    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-vr858"
Jul 16 21:46:55.326964 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.326816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:46:55.332686 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.332666    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8"
Jul 16 21:46:55.340788 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.340770    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-vvs4k"
Jul 16 21:46:55.346505 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.346485    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gq6wg"
Jul 16 21:46:55.353302 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.353282    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:46:55.360252 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.360192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-kqtnz"
Jul 16 21:46:55.363901 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.363875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-q7lj4"
Jul 16 21:46:55.499769 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.499729    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e58abf2_4c9c_414f_9fc1_bf1a4b41b4cc.slice/crio-a400d73d2b832ea47b25e002ddbfe1d193da0b22ca2e148e58dbd7116bcda32e WatchSource:0}: Error finding container a400d73d2b832ea47b25e002ddbfe1d193da0b22ca2e148e58dbd7116bcda32e: Status 404 returned error can't find the container with id a400d73d2b832ea47b25e002ddbfe1d193da0b22ca2e148e58dbd7116bcda32e
Jul 16 21:46:55.501601 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.501564    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3432cb8b_9202_4e9a_a1de_d26494fa415b.slice/crio-06e84a40e81319200c06728b952787739de6ad32afa0dd2c04f99b16fea8487c WatchSource:0}: Error finding container 06e84a40e81319200c06728b952787739de6ad32afa0dd2c04f99b16fea8487c: Status 404 returned error can't find the container with id 06e84a40e81319200c06728b952787739de6ad32afa0dd2c04f99b16fea8487c
Jul 16 21:46:55.523118 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.523095    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod383e7ef0_501b_416f_ba30_712db4ed855b.slice/crio-4b754cb0988c756bdc6ac55e663ee25e38e0abdb5bd4bfa8d9c3ac5b48cb8b76 WatchSource:0}: Error finding container 4b754cb0988c756bdc6ac55e663ee25e38e0abdb5bd4bfa8d9c3ac5b48cb8b76: Status 404 returned error can't find the container with id 4b754cb0988c756bdc6ac55e663ee25e38e0abdb5bd4bfa8d9c3ac5b48cb8b76
Jul 16 21:46:55.525656 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.525634    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7befa2ae_6b26_46a9_a769_de9e01d7c4b3.slice/crio-8ff9d3e18ab878e9e01081573b5d2dfbe60a5f5ceb3558de2f7cf8784c19b712 WatchSource:0}: Error finding container 8ff9d3e18ab878e9e01081573b5d2dfbe60a5f5ceb3558de2f7cf8784c19b712: Status 404 returned error can't find the container with id 8ff9d3e18ab878e9e01081573b5d2dfbe60a5f5ceb3558de2f7cf8784c19b712
Jul 16 21:46:55.526544 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.526426    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda599a220_6fa7_4d9e_be51_52258ab5f6c0.slice/crio-8e7833092a6e16fa694261b5912c50c048b23b650da6da7d8e951c7ff6cd4066 WatchSource:0}: Error finding container 8e7833092a6e16fa694261b5912c50c048b23b650da6da7d8e951c7ff6cd4066: Status 404 returned error can't find the container with id 8e7833092a6e16fa694261b5912c50c048b23b650da6da7d8e951c7ff6cd4066
Jul 16 21:46:55.527775 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.527670    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb68d5ac_5e6e_434e_9bff_7c296bff4a90.slice/crio-7d1bb0b8632810a4acd17fdd1821c54fbd3b8dfb7d7b4ed6c7316fbcd0ab7eff WatchSource:0}: Error finding container 7d1bb0b8632810a4acd17fdd1821c54fbd3b8dfb7d7b4ed6c7316fbcd0ab7eff: Status 404 returned error can't find the container with id 7d1bb0b8632810a4acd17fdd1821c54fbd3b8dfb7d7b4ed6c7316fbcd0ab7eff
Jul 16 21:46:55.528133 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.528110    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8713e00_9c4c_419a_bc37_20954386ea69.slice/crio-eada9ec4ac8bb714bf45ecd2f9795097b3675dad05638ce956e8cbfa712e0d07 WatchSource:0}: Error finding container eada9ec4ac8bb714bf45ecd2f9795097b3675dad05638ce956e8cbfa712e0d07: Status 404 returned error can't find the container with id eada9ec4ac8bb714bf45ecd2f9795097b3675dad05638ce956e8cbfa712e0d07
Jul 16 21:46:55.528905 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.528886    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfff8bdec_cd9f_413d_8d0a_9b5fdb06b759.slice/crio-6fab73ea44b1e292a72210e40c30384a1df46499691d1c478a6ff5d76138f6a7 WatchSource:0}: Error finding container 6fab73ea44b1e292a72210e40c30384a1df46499691d1c478a6ff5d76138f6a7: Status 404 returned error can't find the container with id 6fab73ea44b1e292a72210e40c30384a1df46499691d1c478a6ff5d76138f6a7
Jul 16 21:46:55.530102 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:46:55.530081    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cc7a5d8_eda1_4e1d_9421_a8933d99c709.slice/crio-bed4fcb9e70ffcb2b3ebc129c091622d044ada7b9e7dd9f3ec8899786f841312 WatchSource:0}: Error finding container bed4fcb9e70ffcb2b3ebc129c091622d044ada7b9e7dd9f3ec8899786f841312: Status 404 returned error can't find the container with id bed4fcb9e70ffcb2b3ebc129c091622d044ada7b9e7dd9f3ec8899786f841312
Jul 16 21:46:55.721394 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.721362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:55.721514 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.721427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:55.721569 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.721520    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:55.721569 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.721528    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:55.721654 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.721570    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:56.721552456 +0000 UTC m=+4.050992343 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:55.721654 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.721594    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:46:56.72157727 +0000 UTC m=+4.051017152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:55.822246 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:55.822038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:55.822246 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.822245    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:46:55.822461 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.822271    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:46:55.822461 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.822281    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:55.822461 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:55.822345    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:56.822325418 +0000 UTC m=+4.151765320 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:56.019093 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.019026    2581 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:46:56.091808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.091742    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:41:55 +0000 UTC" deadline="2028-01-15 21:01:59.335841252 +0000 UTC"
Jul 16 21:46:56.091808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.091804    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13151h15m3.244041618s"
Jul 16 21:46:56.154931 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.154896    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:56.155097 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.155021    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:46:56.155535 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.155511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:56.155658 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.155621    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:46:56.168116 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.167327    2581 generic.go:356] "Generic (PLEG): container finished" podID="9234415a6de9d755ed5f51d746845080" containerID="9b41f658fd392bef5f024fe62e9dff05b16c35ab72b2a698b46ffb735c87899b" exitCode=0
Jul 16 21:46:56.168116 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.167867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9234415a6de9d755ed5f51d746845080","Type":"ContainerDied","Data":"9b41f658fd392bef5f024fe62e9dff05b16c35ab72b2a698b46ffb735c87899b"}
Jul 16 21:46:56.174339 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.174285    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqtnz" event={"ID":"cb68d5ac-5e6e-434e-9bff-7c296bff4a90","Type":"ContainerStarted","Data":"7d1bb0b8632810a4acd17fdd1821c54fbd3b8dfb7d7b4ed6c7316fbcd0ab7eff"}
Jul 16 21:46:56.182496 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.182399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ss5km" event={"ID":"383e7ef0-501b-416f-ba30-712db4ed855b","Type":"ContainerStarted","Data":"4b754cb0988c756bdc6ac55e663ee25e38e0abdb5bd4bfa8d9c3ac5b48cb8b76"}
Jul 16 21:46:56.186115 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.186085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8" event={"ID":"3432cb8b-9202-4e9a-a1de-d26494fa415b","Type":"ContainerStarted","Data":"06e84a40e81319200c06728b952787739de6ad32afa0dd2c04f99b16fea8487c"}
Jul 16 21:46:56.188722 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.188699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gq6wg" event={"ID":"0cc7a5d8-eda1-4e1d-9421-a8933d99c709","Type":"ContainerStarted","Data":"bed4fcb9e70ffcb2b3ebc129c091622d044ada7b9e7dd9f3ec8899786f841312"}
Jul 16 21:46:56.190288 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.190264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vr858" event={"ID":"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759","Type":"ContainerStarted","Data":"6fab73ea44b1e292a72210e40c30384a1df46499691d1c478a6ff5d76138f6a7"}
Jul 16 21:46:56.191766 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.191742    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerStarted","Data":"eada9ec4ac8bb714bf45ecd2f9795097b3675dad05638ce956e8cbfa712e0d07"}
Jul 16 21:46:56.193032 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.193008    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7lj4" event={"ID":"a599a220-6fa7-4d9e-be51-52258ab5f6c0","Type":"ContainerStarted","Data":"8e7833092a6e16fa694261b5912c50c048b23b650da6da7d8e951c7ff6cd4066"}
Jul 16 21:46:56.198598 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.198505    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"8ff9d3e18ab878e9e01081573b5d2dfbe60a5f5ceb3558de2f7cf8784c19b712"}
Jul 16 21:46:56.203246 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.200642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xsfvt" event={"ID":"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc","Type":"ContainerStarted","Data":"a400d73d2b832ea47b25e002ddbfe1d193da0b22ca2e148e58dbd7116bcda32e"}
Jul 16 21:46:56.729385 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.729355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:56.729495 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.729427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:56.729559 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.729544    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:56.729619 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.729608    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:46:58.729590583 +0000 UTC m=+6.059030469 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:56.729974 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.729958    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:56.730059 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.730005    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:58.729991347 +0000 UTC m=+6.059431231 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:56.830643 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:56.830011    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:56.830643 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.830243    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:46:56.830643 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.830266    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:46:56.830643 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.830278    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:56.830643 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:56.830333    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:46:58.830314975 +0000 UTC m=+6.159754859 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:57.092319 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.092208    2581 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:41:55 +0000 UTC" deadline="2027-12-27 19:39:13.48332347 +0000 UTC"
Jul 16 21:46:57.092319 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.092259    2581 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12693h52m16.391069213s"
Jul 16 21:46:57.157514 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.157345    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:57.157514 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:57.157475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:46:57.212834 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.212164    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" event={"ID":"ec3998bf518cc797a77d2a84d9ceb141","Type":"ContainerStarted","Data":"2bcc64240cc3df04e74df5535590c8f17e041dff81eb7161dc8467e3dca49678"}
Jul 16 21:46:57.219851 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.219374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" event={"ID":"9234415a6de9d755ed5f51d746845080","Type":"ContainerStarted","Data":"3af82409b4bc6cf6f9e5bb201109d5bd66f72ad20fd93d7d1ed872751c778151"}
Jul 16 21:46:57.251677 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:57.250961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-223.ec2.internal" podStartSLOduration=3.250940099 podStartE2EDuration="3.250940099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:57.231556853 +0000 UTC m=+4.560996759" watchObservedRunningTime="2026-07-16 21:46:57.250940099 +0000 UTC m=+4.580380003"
Jul 16 21:46:58.154790 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:58.154758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:58.155267 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.154887    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:46:58.155389 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:58.155344    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:58.155470 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.155449    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:46:58.746511 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:58.746468    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:46:58.746673 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:58.746538    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:46:58.746673 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.746657    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:58.746673 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.746660    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:58.746909 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.746723    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:02.746707 +0000 UTC m=+10.076146887 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:46:58.746909 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.746740    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:02.746734327 +0000 UTC m=+10.076174208 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:46:58.847794 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:58.847759    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:58.847960 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.847904    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:46:58.847960 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.847925    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:46:58.847960 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.847936    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:58.848126 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:58.847994    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:02.847975074 +0000 UTC m=+10.177414977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:46:59.157559 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:46:59.157491    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:46:59.157881 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:46:59.157616    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:00.155062 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:00.154578    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:00.155062 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:00.154595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:00.155062 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:00.154714    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:00.155062 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:00.154758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:01.157433 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:01.157396    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:01.157827 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:01.157525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:02.155334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:02.154691    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:02.155334 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.154818    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:02.155334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:02.155149    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:02.155334 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.155291    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:02.781255 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:02.781093    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:02.781255 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:02.781179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:02.781758 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.781290    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:02.781758 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.781313    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:02.781758 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.781366    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:10.781344149 +0000 UTC m=+18.110784048 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:02.781758 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.781389    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:10.781377695 +0000 UTC m=+18.110817582 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:02.882151 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:02.882113    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:02.882331 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.882287    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:02.882331 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.882309    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:02.882331 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.882321    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:02.882474 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:02.882385    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:10.882366152 +0000 UTC m=+18.211806038 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:03.155650 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:03.155072    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:03.155650 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:03.155184    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:04.154834 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:04.154758    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:04.155287 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:04.154873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:04.155287 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:04.155254    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:04.155396 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:04.155358    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:05.158692 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:05.158659    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:05.159048 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:05.158785    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:06.154459 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:06.154419    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:06.154627 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:06.154469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:06.154627 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:06.154575    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:06.154715 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:06.154693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:07.154547 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:07.154511    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:07.155004 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:07.154640    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:08.154981 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:08.154938    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:08.155524 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:08.154945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:08.155524 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:08.155072    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:08.155524 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:08.155153    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:09.155089 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:09.155052    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:09.155513 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:09.155180    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:10.154474 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:10.154432    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:10.154698 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:10.154442    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:10.154698 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.154573    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:10.154698 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.154654    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:10.844956 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:10.844914    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:10.845570 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:10.845012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:10.845570 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.845093    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:10.845570 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.845111    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:10.845570 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.845168    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:26.845150928 +0000 UTC m=+34.174590808 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:10.845570 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.845194    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:26.845174473 +0000 UTC m=+34.174614376 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:10.945509 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:10.945470    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:10.945720 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.945675    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:10.945720 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.945702    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:10.945720 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.945717    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:10.945875 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:10.945788    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:26.945767926 +0000 UTC m=+34.275207819 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:11.155268 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:11.155186    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:11.155427 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:11.155308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:12.154457 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:12.154421    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:12.154938 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:12.154425    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:12.154938 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:12.154558    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:12.154938 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:12.154693    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:13.155356 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:13.155322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:13.155770 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:13.155435    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:14.155270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:14.155222    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:14.155460 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:14.155382    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:14.155770 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:14.155451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:14.155770 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:14.155559    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:15.155399 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.155162    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:15.155519 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:15.155498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:15.263114 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.263082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-kqtnz" event={"ID":"cb68d5ac-5e6e-434e-9bff-7c296bff4a90","Type":"ContainerStarted","Data":"3e3845ba7698c3686de481fc577fc07447a0f64a5ac6710d03b1c7ae5c9738ec"}
Jul 16 21:47:15.264864 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.264826    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-ss5km" event={"ID":"383e7ef0-501b-416f-ba30-712db4ed855b","Type":"ContainerStarted","Data":"c92e6365d39588dfe04f1365ea703e047aabaf93fddb69b1d7d32b8df513718f"}
Jul 16 21:47:15.266502 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.266470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8" event={"ID":"3432cb8b-9202-4e9a-a1de-d26494fa415b","Type":"ContainerStarted","Data":"6321f327083c54b3576fef979600f0202eb5b4b5f351346c541ff2da920f3522"}
Jul 16 21:47:15.268251 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.268204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gq6wg" event={"ID":"0cc7a5d8-eda1-4e1d-9421-a8933d99c709","Type":"ContainerStarted","Data":"c642cda0c00ee13957fa75a50c118757aa366add17a1ac2bf1ff6bd224bc539b"}
Jul 16 21:47:15.269834 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.269804    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-vr858" event={"ID":"fff8bdec-cd9f-413d-8d0a-9b5fdb06b759","Type":"ContainerStarted","Data":"3b91b1e82a042685b881aaf74c2ee671dd8b9d8fb621707b88c67d4af5a73479"}
Jul 16 21:47:15.271489 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.271465    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerStarted","Data":"65df5699f216103634f95c0e737cc5deb2ab9484f6775b7be3c4328a8c908f0c"}
Jul 16 21:47:15.272981 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.272959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-q7lj4" event={"ID":"a599a220-6fa7-4d9e-be51-52258ab5f6c0","Type":"ContainerStarted","Data":"5a7e7bdeed6b43cac449745f98c59d424ad929857b3fc9c34e8c8ec4977abef9"}
Jul 16 21:47:15.286925 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.286876    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-223.ec2.internal" podStartSLOduration=21.286859095 podStartE2EDuration="21.286859095s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:46:57.251646491 +0000 UTC m=+4.581086396" watchObservedRunningTime="2026-07-16 21:47:15.286859095 +0000 UTC m=+22.616299002"
Jul 16 21:47:15.287240 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.287196    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-kqtnz" podStartSLOduration=2.948964793 podStartE2EDuration="22.28718917s" totalImagesPullingTime="19.338224377s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:15.286454006 +0000 UTC m=+22.615893910" watchObservedRunningTime="2026-07-16 21:47:15.28718917 +0000 UTC m=+22.616629074"
Jul 16 21:47:15.305844 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.305798    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-vr858" podStartSLOduration=2.939627227 podStartE2EDuration="22.305784313s" totalImagesPullingTime="19.366157086s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:15.305514362 +0000 UTC m=+22.634954278" watchObservedRunningTime="2026-07-16 21:47:15.305784313 +0000 UTC m=+22.635224214"
Jul 16 21:47:15.321884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.321836    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gq6wg" podStartSLOduration=2.985531848 podStartE2EDuration="22.32181685s" totalImagesPullingTime="19.336285002s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:15.321411084 +0000 UTC m=+22.650850986" watchObservedRunningTime="2026-07-16 21:47:15.32181685 +0000 UTC m=+22.651256753"
Jul 16 21:47:15.337150 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:15.337097    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-q7lj4" podStartSLOduration=3.001264164 podStartE2EDuration="22.33708162s" totalImagesPullingTime="19.335817456s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:15.33672238 +0000 UTC m=+22.666162339" watchObservedRunningTime="2026-07-16 21:47:15.33708162 +0000 UTC m=+22.666521522"
Jul 16 21:47:16.154933 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.154753    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:16.155039 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.154780    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:16.155039 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:16.155022    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:16.155115 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:16.155081    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:16.277456 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.277424    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="65df5699f216103634f95c0e737cc5deb2ab9484f6775b7be3c4328a8c908f0c" exitCode=0
Jul 16 21:47:16.278177 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.277495    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"65df5699f216103634f95c0e737cc5deb2ab9484f6775b7be3c4328a8c908f0c"}
Jul 16 21:47:16.284654 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.284636    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:47:16.284950 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.284934    2581 generic.go:356] "Generic (PLEG): container finished" podID="7befa2ae-6b26-46a9-a769-de9e01d7c4b3" containerID="c1be2a69c8d136e14b0f2af0cb27cde497a7305b28bd4cc5e1fd0390b0bb9aba" exitCode=1
Jul 16 21:47:16.285003 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.284991    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"ab62ea47446b9d782f69242123cef81ef60569eed3233fa7261fe471286c2a17"}
Jul 16 21:47:16.285056 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.285046    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"df9940922d6c6fc709bc5662d620c8f1c399158c951d9525f50a06e48420cfad"}
Jul 16 21:47:16.285090 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.285080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"e5c80f67395494c9dada94328737037a3e62926987e77146c881ab13dbd3a21a"}
Jul 16 21:47:16.285121 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.285110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerDied","Data":"c1be2a69c8d136e14b0f2af0cb27cde497a7305b28bd4cc5e1fd0390b0bb9aba"}
Jul 16 21:47:16.285156 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.285145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"0f489a7918163d68c3ba6dedcdf64628ecff694bbfebf3619366d0d39a529a67"}
Jul 16 21:47:16.287952 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.287910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-xsfvt" event={"ID":"1e58abf2-4c9c-414f-9fc1-bf1a4b41b4cc","Type":"ContainerStarted","Data":"3a28d1f48de08810c4b7e99ee6acc8c14bbf8736ae3feb2ea7c42069e356b61b"}
Jul 16 21:47:16.300740 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.300696    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-ss5km" podStartSLOduration=12.324579183 podStartE2EDuration="23.300681816s" totalImagesPullingTime="10.976102633s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:15.380120883 +0000 UTC m=+22.709560786" watchObservedRunningTime="2026-07-16 21:47:16.300681816 +0000 UTC m=+23.630121719"
Jul 16 21:47:16.372662 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:16.372643    2581 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:47:17.080114 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.079294    2581 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:47:16.372657637Z","UUID":"b45752bb-f8ae-47a9-a792-15785ba582cb","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:47:17.081742 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.081250    2581 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:47:17.081742 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.081296    2581 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:47:17.154839 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.154803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:17.155056 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:17.154944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:17.293356 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.293326    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:47:17.293920 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.293810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"2e31c399afbf0cd1051aea75dc9e2f0255cdbd4a4d181c5b4d1ef018c6288f1f"}
Jul 16 21:47:17.295796 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:17.295765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8" event={"ID":"3432cb8b-9202-4e9a-a1de-d26494fa415b","Type":"ContainerStarted","Data":"adce7e34cf06fbf8c09a26d366d3c60256b26d2340e5d5403617113ec4ff2331"}
Jul 16 21:47:18.154528 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.154492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:18.154788 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.154492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:18.154788 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:18.154623    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:18.154788 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:18.154707    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:18.301039 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.301004    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8" event={"ID":"3432cb8b-9202-4e9a-a1de-d26494fa415b","Type":"ContainerStarted","Data":"eaaad0ea569427b613a61fa19ce5664821f66761b4ab052741cb53de0b7cc2b8"}
Jul 16 21:47:18.335138 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.335074    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-xsfvt" podStartSLOduration=5.9704149520000005 podStartE2EDuration="25.335045616s" totalImagesPullingTime="19.364630664s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:16.316398451 +0000 UTC m=+23.645838353" watchObservedRunningTime="2026-07-16 21:47:18.335045616 +0000 UTC m=+25.664485520"
Jul 16 21:47:18.573660 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.573623    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:47:18.574161 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.574142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:47:18.591334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:18.591284    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-86qp8" podStartSLOduration=12.196851991 podStartE2EDuration="25.591268294s" totalImagesPullingTime="13.394416303s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:18.336185601 +0000 UTC m=+25.665625504" watchObservedRunningTime="2026-07-16 21:47:18.591268294 +0000 UTC m=+25.920708197"
Jul 16 21:47:19.155425 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:19.155175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:19.155643 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:19.155524    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:19.306891 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:19.306861    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:47:19.307385 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:19.307347    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"b350dabdb33366408c496a599b24e92661cd879d7a0ab387c2b411d8c93ffb5c"}
Jul 16 21:47:19.307709 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:19.307683    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:47:19.308105 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:19.308084    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-ss5km"
Jul 16 21:47:20.155240 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:20.155193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:20.155450 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:20.155302    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:20.155450 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:20.155365    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:20.155561 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:20.155482    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:21.154901 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:21.154878    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:21.155721 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:21.155002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:21.315926 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:21.315758    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:47:21.316283 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:21.316261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"739e9e4b8cbe78d0aa89aad757cc0150f594260d8dbb6c9186a2c6e5c69f6ba3"}
Jul 16 21:47:21.316673 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:21.316656    2581 scope.go:117] "RemoveContainer" containerID="c1be2a69c8d136e14b0f2af0cb27cde497a7305b28bd4cc5e1fd0390b0bb9aba"
Jul 16 21:47:22.154833 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.154803    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:22.154989 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:22.154920    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:22.155264 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.154982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:22.155264 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:22.155073    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:22.321670 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.321570    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="360543c369ff8a810db97b6cd4ecbb1765f13f9df53731ea82534c3922b7ae0d" exitCode=0
Jul 16 21:47:22.321848 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.321674    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"360543c369ff8a810db97b6cd4ecbb1765f13f9df53731ea82534c3922b7ae0d"}
Jul 16 21:47:22.328936 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.328908    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:47:22.329350 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.329317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" event={"ID":"7befa2ae-6b26-46a9-a769-de9e01d7c4b3","Type":"ContainerStarted","Data":"d48fdae84594be1b5b734488fc96362887ad946725d14440aeb87421c87fbaa5"}
Jul 16 21:47:22.329667 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.329631    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:22.329753 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.329726    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:22.329804 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.329771    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:22.346874 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.346845    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:22.347380 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.347358    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:22.394083 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:22.394018    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6" podStartSLOduration=9.458536049 podStartE2EDuration="29.393997768s" totalImagesPullingTime="19.935461719s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:22.392665962 +0000 UTC m=+29.722105871" watchObservedRunningTime="2026-07-16 21:47:22.393997768 +0000 UTC m=+29.723437672"
Jul 16 21:47:23.155391 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.155359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:23.155858 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:23.155497    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:23.454410 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.454347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqkd4"]
Jul 16 21:47:23.454534 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.454478    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:23.454572 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:23.454559    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:23.457151 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.457127    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qtz7s"]
Jul 16 21:47:23.457339 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.457215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:23.457339 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:23.457322    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:23.457893 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.457870    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxzbc"]
Jul 16 21:47:23.457973 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:23.457954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:23.458057 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:23.458039    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:24.335610 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:24.335422    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="733902e10676a61c8187a797c2c436f861c8201ea868a68f6ea23809b45d0bcf" exitCode=0
Jul 16 21:47:24.335965 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:24.335502    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"733902e10676a61c8187a797c2c436f861c8201ea868a68f6ea23809b45d0bcf"}
Jul 16 21:47:25.155149 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:25.155119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:25.155149 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:25.155130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:25.155368 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:25.155255    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:25.155368 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:25.155345    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:25.155484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:25.155405    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:25.155532 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:25.155492    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:26.341378 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:26.341347    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="b36b486d67ae59a3a61dd0e59e5fe3d24175ba91d396da7966d37facfd7989d6" exitCode=0
Jul 16 21:47:26.341817 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:26.341397    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"b36b486d67ae59a3a61dd0e59e5fe3d24175ba91d396da7966d37facfd7989d6"}
Jul 16 21:47:26.859720 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:26.859682    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:26.859906 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:26.859753    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:26.859906 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.859829    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:26.859906 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.859844    2581 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:26.859906 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.859897    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret podName:fca4ff45-8365-4629-88a8-e67e2f1bfc7d nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.859882095 +0000 UTC m=+66.189321979 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret") pod "global-pull-secret-syncer-zqkd4" (UID: "fca4ff45-8365-4629-88a8-e67e2f1bfc7d") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:47:26.860078 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.859912    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.859905135 +0000 UTC m=+66.189345015 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:47:26.960734 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:26.960697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:26.960925 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.960890    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:47:26.960925 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.960915    2581 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:47:26.960925 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.960924    2581 projected.go:196] Error preparing data for projected volume kube-api-access-g7q7h for pod openshift-network-diagnostics/network-check-target-qtz7s: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:26.961038 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:26.960981    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h podName:8a6e7ba6-3a88-4536-a946-4c9fceb807f9 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:58.960962198 +0000 UTC m=+66.290402099 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-g7q7h" (UniqueName: "kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h") pod "network-check-target-qtz7s" (UID: "8a6e7ba6-3a88-4536-a946-4c9fceb807f9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:47:27.154824 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:27.154731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:27.154981 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:27.154859    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:27.154981 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:27.154869    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:47:27.154981 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:27.154884    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:27.154981 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:27.154953    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zqkd4" podUID="fca4ff45-8365-4629-88a8-e67e2f1bfc7d"
Jul 16 21:47:27.155188 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:27.155018    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qtz7s" podUID="8a6e7ba6-3a88-4536-a946-4c9fceb807f9"
Jul 16 21:47:28.963736 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:28.963665    2581 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-223.ec2.internal" event="NodeReady"
Jul 16 21:47:28.964268 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:28.963808    2581 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:47:29.015426 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.015391    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-fx9mf"]
Jul 16 21:47:29.018617 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.018596    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-7prpt"]
Jul 16 21:47:29.018728 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.018715    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.021697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.021676    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:47:29.021812 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.021729    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-275zl\""
Jul 16 21:47:29.021812 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.021786    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:47:29.022148 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.022130    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.024292 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.024273    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:47:29.024403 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.024291    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:47:29.027209 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.027195    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7z9z2\""
Jul 16 21:47:29.027533 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.027512    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:47:29.033125 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.033101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7prpt"]
Jul 16 21:47:29.036265 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.034629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fx9mf"]
Jul 16 21:47:29.155086 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.155049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:29.155086 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.155065    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:29.155355 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.155062    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:29.158324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158109    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:47:29.158324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158181    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wc4rs\""
Jul 16 21:47:29.158324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158204    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gkkkc\""
Jul 16 21:47:29.158324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158211    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:47:29.158324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158261    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:47:29.158633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.158361    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:47:29.177483 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177460    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7b07-a130-4366-b420-03897d2ee326-tmp-dir\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.177578 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177486    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd7b07-a130-4366-b420-03897d2ee326-config-volume\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.177578 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r8dj\" (UniqueName: \"kubernetes.io/projected/12cd7b07-a130-4366-b420-03897d2ee326-kube-api-access-5r8dj\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.177645 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177580    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe2c42b-98c5-491d-b75d-a817769e7c85-cert\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.177645 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.177645 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.177632    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vkph\" (UniqueName: \"kubernetes.io/projected/9fe2c42b-98c5-491d-b75d-a817769e7c85-kube-api-access-4vkph\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.278955 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.278874    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vkph\" (UniqueName: \"kubernetes.io/projected/9fe2c42b-98c5-491d-b75d-a817769e7c85-kube-api-access-4vkph\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.278955 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.278927    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7b07-a130-4366-b420-03897d2ee326-tmp-dir\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.278955 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.278948    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd7b07-a130-4366-b420-03897d2ee326-config-volume\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.279214 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.278975    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r8dj\" (UniqueName: \"kubernetes.io/projected/12cd7b07-a130-4366-b420-03897d2ee326-kube-api-access-5r8dj\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.279214 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.279031    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe2c42b-98c5-491d-b75d-a817769e7c85-cert\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.279214 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.279072    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.279214 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:29.279167    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:29.279214 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:29.279212    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:29.779197888 +0000 UTC m=+37.108637769 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:29.279506 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.279339    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12cd7b07-a130-4366-b420-03897d2ee326-tmp-dir\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.279794 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.279768    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12cd7b07-a130-4366-b420-03897d2ee326-config-volume\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.283996 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.283975    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/9fe2c42b-98c5-491d-b75d-a817769e7c85-cert\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.289160 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.289137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vkph\" (UniqueName: \"kubernetes.io/projected/9fe2c42b-98c5-491d-b75d-a817769e7c85-kube-api-access-4vkph\") pod \"ingress-canary-7prpt\" (UID: \"9fe2c42b-98c5-491d-b75d-a817769e7c85\") " pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.292679 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.292650    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r8dj\" (UniqueName: \"kubernetes.io/projected/12cd7b07-a130-4366-b420-03897d2ee326-kube-api-access-5r8dj\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.338759 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.338721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-7prpt"
Jul 16 21:47:29.485013 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.484833    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-7prpt"]
Jul 16 21:47:29.490273 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:47:29.490220    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fe2c42b_98c5_491d_b75d_a817769e7c85.slice/crio-95d99e71b29fbc9dd38280cb4dc92a39529e0363752e6017c49230b246c52891 WatchSource:0}: Error finding container 95d99e71b29fbc9dd38280cb4dc92a39529e0363752e6017c49230b246c52891: Status 404 returned error can't find the container with id 95d99e71b29fbc9dd38280cb4dc92a39529e0363752e6017c49230b246c52891
Jul 16 21:47:29.782657 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:29.782619    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:29.782840 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:29.782759    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:29.782840 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:29.782835    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:30.782815587 +0000 UTC m=+38.112255468 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:30.352115 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:30.352068    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7prpt" event={"ID":"9fe2c42b-98c5-491d-b75d-a817769e7c85","Type":"ContainerStarted","Data":"95d99e71b29fbc9dd38280cb4dc92a39529e0363752e6017c49230b246c52891"}
Jul 16 21:47:30.791762 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:30.791675    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:30.791902 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:30.791842    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:30.791952 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:30.791910    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:32.791891811 +0000 UTC m=+40.121331694 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:32.360015 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:32.359982    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="9f881a145a42b18da034c0f9a9c32741943fdee2dd8d36c76e602c7d57ddf1b4" exitCode=0
Jul 16 21:47:32.360525 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:32.360041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"9f881a145a42b18da034c0f9a9c32741943fdee2dd8d36c76e602c7d57ddf1b4"}
Jul 16 21:47:32.807507 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:32.807424    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:32.807652 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:32.807577    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:32.807718 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:32.807655    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:36.807635186 +0000 UTC m=+44.137075070 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:33.366909 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:33.366869    2581 generic.go:356] "Generic (PLEG): container finished" podID="a8713e00-9c4c-419a-bc37-20954386ea69" containerID="8ef4a0e11e3e25c10bcdfb0a8f1acd0e1aa28569777d41e1ab3ab1db8f672903" exitCode=0
Jul 16 21:47:33.367335 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:33.366941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerDied","Data":"8ef4a0e11e3e25c10bcdfb0a8f1acd0e1aa28569777d41e1ab3ab1db8f672903"}
Jul 16 21:47:34.371802 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:34.371769    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" event={"ID":"a8713e00-9c4c-419a-bc37-20954386ea69","Type":"ContainerStarted","Data":"20613e762a0ef89573d5391cb55594c365a8ac3351508162db0232ce214301f1"}
Jul 16 21:47:34.373019 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:34.372995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-7prpt" event={"ID":"9fe2c42b-98c5-491d-b75d-a817769e7c85","Type":"ContainerStarted","Data":"38ba1c7bd04dafd6c416faa01262552e4ec0d352f2d510ab385df0ca2cd591e8"}
Jul 16 21:47:34.398161 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:34.398119    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-vvs4k" podStartSLOduration=9.594469034 podStartE2EDuration="41.398104997s" totalImagesPullingTime="31.803635963s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:34.39646246 +0000 UTC m=+41.725902364" watchObservedRunningTime="2026-07-16 21:47:34.398104997 +0000 UTC m=+41.727544899"
Jul 16 21:47:36.835910 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:36.835872    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:36.836389 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:36.835991    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:36.836389 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:36.836063    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:47:44.836040884 +0000 UTC m=+52.165480772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:44.895114 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:44.895074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:47:44.895648 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:44.895221    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:47:44.895648 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:44.895309    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:00.895287438 +0000 UTC m=+68.224727318 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:47:47.906210 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:47.906157    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-7prpt" podStartSLOduration=15.513695647 podStartE2EDuration="19.906140865s" totalImagesPullingTime="4.392445218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:34.41594907 +0000 UTC m=+41.745388984" watchObservedRunningTime="2026-07-16 21:47:47.906140865 +0000 UTC m=+55.235580745"
Jul 16 21:47:47.906689 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:47.906499    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"]
Jul 16 21:47:47.924304 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:47.924276    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:47.946790 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.946759    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-ca\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\"" type="*v1.Secret"
Jul 16 21:47:47.946790 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.946773    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"kube-root-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap"
Jul 16 21:47:47.946949 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.946761    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" podUID="1799669f-da56-435c-a929-ac4b1273400b" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:47.949089 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.949066    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-hub-kubeconfig\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\"" type="*v1.Secret"
Jul 16 21:47:47.962448 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.962420    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.ConfigMap: configmaps \"openshift-service-ca.crt\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"configmaps\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap"
Jul 16 21:47:47.966853 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.966836    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"open-cluster-management-image-pull-credentials\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\"" type="*v1.Secret"
Jul 16 21:47:47.967111 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.967094    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-service-proxy-server-certificates\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\"" type="*v1.Secret"
Jul 16 21:47:47.970258 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:47.969077    2581 reflector.go:205] "Failed to watch" err="failed to list *v1.Secret: secrets \"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot list resource \"secrets\" in API group \"\" in the namespace \"open-cluster-management-agent-addon\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" logger="UnhandledError" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\"" type="*v1.Secret"
Jul 16 21:47:47.984966 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:47.984937    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"]
Jul 16 21:47:48.018884 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.018856    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.018991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.018893    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qhvx\" (UniqueName: \"kubernetes.io/projected/1799669f-da56-435c-a929-ac4b1273400b-kube-api-access-2qhvx\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.018991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.018962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.019063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.018996    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.019063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.019026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1799669f-da56-435c-a929-ac4b1273400b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.019063 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.019050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119527 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119534    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qhvx\" (UniqueName: \"kubernetes.io/projected/1799669f-da56-435c-a929-ac4b1273400b-kube-api-access-2qhvx\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119590    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119615    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119632    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1799669f-da56-435c-a929-ac4b1273400b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.119683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.119659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.767042 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.767005    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 16 21:47:48.773690 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.773669    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.798683 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.798660    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 16 21:47:48.798784 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.798660    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 16 21:47:48.803019 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.802994    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.988521 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.988490    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 16 21:47:48.990760 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.990732    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1799669f-da56-435c-a929-ac4b1273400b-additional-service-ca\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:48.994710 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:48.994691    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qhvx\" (UniqueName: \"kubernetes.io/projected/1799669f-da56-435c-a929-ac4b1273400b-kube-api-access-2qhvx\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:49.045890 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.045822    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 16 21:47:49.052045 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.052028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-hub\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:49.096068 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.096041    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 16 21:47:49.102571 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.102542    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1799669f-da56-435c-a929-ac4b1273400b-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-55db467cc7-zr6mk\" (UID: \"1799669f-da56-435c-a929-ac4b1273400b\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:49.401863 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.401839    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 16 21:47:49.403935 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.403920    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:47:49.536467 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:49.536438    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"]
Jul 16 21:47:49.539589 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:47:49.539565    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1799669f_da56_435c_a929_ac4b1273400b.slice/crio-0fc6c9d4c501fb3acc75f69fda9d68bd28a67adbbc2fb5c6f0d0d188a9b2fc83 WatchSource:0}: Error finding container 0fc6c9d4c501fb3acc75f69fda9d68bd28a67adbbc2fb5c6f0d0d188a9b2fc83: Status 404 returned error can't find the container with id 0fc6c9d4c501fb3acc75f69fda9d68bd28a67adbbc2fb5c6f0d0d188a9b2fc83
Jul 16 21:47:50.408094 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:50.408053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerStarted","Data":"0fc6c9d4c501fb3acc75f69fda9d68bd28a67adbbc2fb5c6f0d0d188a9b2fc83"}
Jul 16 21:47:53.418120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:53.418084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerStarted","Data":"e18043f52d29012c74b0508b51dc517f871e5f68454285300ece06e3c69c83bc"}
Jul 16 21:47:53.418529 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:53.418137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerStarted","Data":"859ad7d0fe329d26c9f98d130947af13d0ee37a2662982fff3bcd4f47a068129"}
Jul 16 21:47:54.349205 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:54.349181    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-79rh6"
Jul 16 21:47:54.422153 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:54.422120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerStarted","Data":"b5b47f3465c644dd15aafdad3d3423937b5a728c9cf29df04a3321b6266292d6"}
Jul 16 21:47:54.443560 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:54.443510    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" podStartSLOduration=3.7127544009999998 podStartE2EDuration="7.443493749s" totalImagesPullingTime="3.730739348s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:47:54.442319231 +0000 UTC m=+61.771759132" watchObservedRunningTime="2026-07-16 21:47:54.443493749 +0000 UTC m=+61.772933667"
Jul 16 21:47:58.904379 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:58.904339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:47:58.904901 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:58.904408    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:58.909079 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:58.909060    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:47:58.909176 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:58.909159    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:47:58.914962 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:58.914946    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:47:58.915049 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:47:58.914999    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:02.914984143 +0000 UTC m=+130.244424023 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : secret "metrics-daemon-secret" not found
Jul 16 21:47:58.917219 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:58.917202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/fca4ff45-8365-4629-88a8-e67e2f1bfc7d-original-pull-secret\") pod \"global-pull-secret-syncer-zqkd4\" (UID: \"fca4ff45-8365-4629-88a8-e67e2f1bfc7d\") " pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:59.005010 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.004976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:59.010179 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.010158    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:47:59.018326 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.018301    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:47:59.029149 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.029118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7q7h\" (UniqueName: \"kubernetes.io/projected/8a6e7ba6-3a88-4536-a946-4c9fceb807f9-kube-api-access-g7q7h\") pod \"network-check-target-qtz7s\" (UID: \"8a6e7ba6-3a88-4536-a946-4c9fceb807f9\") " pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:59.167886 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.167808    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wc4rs\""
Jul 16 21:47:59.174585 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.174567    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zqkd4"
Jul 16 21:47:59.175670 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.175646    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:47:59.317325 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.317294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zqkd4"]
Jul 16 21:47:59.320929 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:47:59.320904    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca4ff45_8365_4629_88a8_e67e2f1bfc7d.slice/crio-bf9c51f7d2bc9e34459b09b2c54403ebe84a11918ba3d4e2d987d469ceb6bc81 WatchSource:0}: Error finding container bf9c51f7d2bc9e34459b09b2c54403ebe84a11918ba3d4e2d987d469ceb6bc81: Status 404 returned error can't find the container with id bf9c51f7d2bc9e34459b09b2c54403ebe84a11918ba3d4e2d987d469ceb6bc81
Jul 16 21:47:59.331081 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.331058    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qtz7s"]
Jul 16 21:47:59.335819 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:47:59.335793    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a6e7ba6_3a88_4536_a946_4c9fceb807f9.slice/crio-18aa00e5256b53f1c8b6d49186adcb215b031731141904f8d30368b11459d4ad WatchSource:0}: Error finding container 18aa00e5256b53f1c8b6d49186adcb215b031731141904f8d30368b11459d4ad: Status 404 returned error can't find the container with id 18aa00e5256b53f1c8b6d49186adcb215b031731141904f8d30368b11459d4ad
Jul 16 21:47:59.435200 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.435112    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qtz7s" event={"ID":"8a6e7ba6-3a88-4536-a946-4c9fceb807f9","Type":"ContainerStarted","Data":"18aa00e5256b53f1c8b6d49186adcb215b031731141904f8d30368b11459d4ad"}
Jul 16 21:47:59.436319 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:47:59.436288    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqkd4" event={"ID":"fca4ff45-8365-4629-88a8-e67e2f1bfc7d","Type":"ContainerStarted","Data":"bf9c51f7d2bc9e34459b09b2c54403ebe84a11918ba3d4e2d987d469ceb6bc81"}
Jul 16 21:48:00.917720 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:00.917676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:48:00.918119 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:48:00.917806    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:00.918119 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:48:00.917857    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:48:32.917842816 +0000 UTC m=+100.247282696 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:04.454212 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:04.454114    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qtz7s" event={"ID":"8a6e7ba6-3a88-4536-a946-4c9fceb807f9","Type":"ContainerStarted","Data":"98cc87866e9477a0eed40e4fcc337ba5f2f81fc3ba051dd04be4d9c7e3f3be72"}
Jul 16 21:48:04.454679 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:04.454216    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:48:04.455518 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:04.455491    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zqkd4" event={"ID":"fca4ff45-8365-4629-88a8-e67e2f1bfc7d","Type":"ContainerStarted","Data":"f0cab4d6aa32d84c2894327f3df2500e3cb290003835438cf1a5632db30755bb"}
Jul 16 21:48:04.481798 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:04.481685    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qtz7s" podStartSLOduration=66.67416344 podStartE2EDuration="1m11.481669891s" totalImagesPullingTime="4.807506451s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:04.480987323 +0000 UTC m=+71.810427224" watchObservedRunningTime="2026-07-16 21:48:04.481669891 +0000 UTC m=+71.811109792"
Jul 16 21:48:04.512686 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:04.512636    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zqkd4" podStartSLOduration=66.68614656 podStartE2EDuration="1m11.512619294s" totalImagesPullingTime="4.826472734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:48:04.512316809 +0000 UTC m=+71.841756711" watchObservedRunningTime="2026-07-16 21:48:04.512619294 +0000 UTC m=+71.842059195"
Jul 16 21:48:32.934037 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:32.933978    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:48:32.934528 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:48:32.934139    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:48:32.934528 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:48:32.934205    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:36.934191835 +0000 UTC m=+164.263631716 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:48:35.460938 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:48:35.460908    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qtz7s"
Jul 16 21:49:02.946611 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:49:02.946560    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:49:02.947104 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:02.946713    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:49:02.947104 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:02.946795    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.946776312 +0000 UTC m=+252.276216200 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : secret "metrics-daemon-secret" not found
Jul 16 21:49:32.029883 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:32.029833    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fx9mf" podUID="12cd7b07-a130-4366-b420-03897d2ee326"
Jul 16 21:49:32.170929 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:32.170875    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:49:32.701168 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:49:32.701128    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:49:36.981529 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:49:36.981484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:49:36.981943 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:36.981625    2581 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:36.981943 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:49:36.981686    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls podName:12cd7b07-a130-4366-b420-03897d2ee326 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:38.981671733 +0000 UTC m=+286.311111614 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls") pod "dns-default-fx9mf" (UID: "12cd7b07-a130-4366-b420-03897d2ee326") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:43.157218 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:49:43.157190    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:51:04.965888 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:04.965857    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:51:04.966373 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:04.965994    2581 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:51:04.966373 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:04.966052    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs podName:cebc9e23-3745-4d0f-b03f-7b59a152c0e4 nodeName:}" failed. No retries permitted until 2026-07-16 21:53:06.966035839 +0000 UTC m=+374.295475720 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs") pod "network-metrics-daemon-sxzbc" (UID: "cebc9e23-3745-4d0f-b03f-7b59a152c0e4") : secret "metrics-daemon-secret" not found
Jul 16 21:51:09.227975 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.227944    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"]
Jul 16 21:51:09.230652 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.230638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.233975 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.233952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-7w9l5\""
Jul 16 21:51:09.234464 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.234447    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 16 21:51:09.235415 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.235398    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 16 21:51:09.247473 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.247453    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"]
Jul 16 21:51:09.295163 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.295140    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.295309 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.295169    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.396460 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.396431    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.396589 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.396462    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.396589 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:09.396578    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:09.396701 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:09.396689    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:09.89666676 +0000 UTC m=+257.226106642 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:09.397126 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.397109    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.900379 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:09.900339    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:09.900546 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:09.900450    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:09.900546 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:09.900508    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:10.900494588 +0000 UTC m=+258.229934469 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:10.905767 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:10.905731    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:10.906135 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:10.905885    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:10.906135 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:10.905952    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:12.905936472 +0000 UTC m=+260.235376354 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:12.921616 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:12.921576    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:12.921995 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:12.921723    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:12.921995 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:12.921801    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:16.921784095 +0000 UTC m=+264.251223978 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:13.867691 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.867661    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2455k"]
Jul 16 21:51:13.870557 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.870541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:13.873182 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.873051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:51:13.873498 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.873482    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:51:13.874022 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.874007    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jvlkz\""
Jul 16 21:51:13.874022 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.874017    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:51:13.874112 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.874051    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:51:13.881285 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:13.881264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2455k"]
Jul 16 21:51:14.030645 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.030606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc30111-495d-430a-9792-1a3021fd6c78-crio-socket\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.031017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.030670    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.031017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.030697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.031017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.030746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8r9j\" (UniqueName: \"kubernetes.io/projected/6bc30111-495d-430a-9792-1a3021fd6c78-kube-api-access-z8r9j\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.031017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.030792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc30111-495d-430a-9792-1a3021fd6c78-data-volume\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8r9j\" (UniqueName: \"kubernetes.io/projected/6bc30111-495d-430a-9792-1a3021fd6c78-kube-api-access-z8r9j\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131343    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc30111-495d-430a-9792-1a3021fd6c78-data-volume\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131382    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc30111-495d-430a-9792-1a3021fd6c78-crio-socket\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131428    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131457    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.131531    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/6bc30111-495d-430a-9792-1a3021fd6c78-crio-socket\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.131570    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.131628    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.631613421 +0000 UTC m=+261.961053302 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.131660 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.131654    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:14.631635558 +0000 UTC m=+261.961075445 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.131876 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.131716    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/6bc30111-495d-430a-9792-1a3021fd6c78-data-volume\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.140280 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.140260    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8r9j\" (UniqueName: \"kubernetes.io/projected/6bc30111-495d-430a-9792-1a3021fd6c78-kube-api-access-z8r9j\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.634311 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.634270    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.634311 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:14.634318    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:14.634510 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.634430    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.634510 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.634449    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:14.634510 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.634496    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:15.634480795 +0000 UTC m=+262.963920676 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:14.634510 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:14.634509    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:15.634503472 +0000 UTC m=+262.963943353 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:15.643982 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:15.643939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:15.643982 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:15.643984    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:15.644416 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:15.644093    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:15.644416 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:15.644097    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:15.644416 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:15.644150    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:17.644137253 +0000 UTC m=+264.973577135 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:15.644416 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:15.644163    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:17.644157098 +0000 UTC m=+264.973596978 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:16.437360 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.437327    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-r2zth"]
Jul 16 21:51:16.440304 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.440289    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.445069 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.445046    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 21:51:16.445176 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.445047    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-vc2zh\""
Jul 16 21:51:16.445579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.445561    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 21:51:16.445680 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.445605    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 21:51:16.445793 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.445781    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 21:51:16.457958 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.457934    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-r2zth"]
Jul 16 21:51:16.549914 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.549883    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8dj2\" (UniqueName: \"kubernetes.io/projected/12a42f6a-35a5-4790-a03a-0a5334058a1d-kube-api-access-v8dj2\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.550110 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.549958    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-cabundle\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.550110 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.550047    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-key\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.650797 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.650767    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-cabundle\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.651184 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.650818    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-key\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.651184 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.650847    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8dj2\" (UniqueName: \"kubernetes.io/projected/12a42f6a-35a5-4790-a03a-0a5334058a1d-kube-api-access-v8dj2\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.651473 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.651453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-cabundle\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.653442 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.653420    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/12a42f6a-35a5-4790-a03a-0a5334058a1d-signing-key\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.660752 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.660730    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8dj2\" (UniqueName: \"kubernetes.io/projected/12a42f6a-35a5-4790-a03a-0a5334058a1d-kube-api-access-v8dj2\") pod \"service-ca-58598695f7-r2zth\" (UID: \"12a42f6a-35a5-4790-a03a-0a5334058a1d\") " pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.749758 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.749656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-r2zth"
Jul 16 21:51:16.856046 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.856021    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gq6wg_0cc7a5d8-eda1-4e1d-9421-a8933d99c709/dns-node-resolver/0.log"
Jul 16 21:51:16.874500 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.874471    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-r2zth"]
Jul 16 21:51:16.878508 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:16.878484    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a42f6a_35a5_4790_a03a_0a5334058a1d.slice/crio-9895571b14dbac55d7a997e4c84ae73d0b179ee7b1bcea7bfbba5a8c0695beb7 WatchSource:0}: Error finding container 9895571b14dbac55d7a997e4c84ae73d0b179ee7b1bcea7bfbba5a8c0695beb7: Status 404 returned error can't find the container with id 9895571b14dbac55d7a997e4c84ae73d0b179ee7b1bcea7bfbba5a8c0695beb7
Jul 16 21:51:16.953082 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.953052    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:16.953246 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:16.953183    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:16.953307 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:16.953265    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:24.953224397 +0000 UTC m=+272.282664278 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:16.984997 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:16.984970    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-r2zth" event={"ID":"12a42f6a-35a5-4790-a03a-0a5334058a1d","Type":"ContainerStarted","Data":"9895571b14dbac55d7a997e4c84ae73d0b179ee7b1bcea7bfbba5a8c0695beb7"}
Jul 16 21:51:17.658012 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:17.657979    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:17.658012 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:17.658017    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:17.658523 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:17.658106    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:17.658523 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:17.658144    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:17.658523 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:17.658174    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.658155596 +0000 UTC m=+268.987595477 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:17.658523 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:17.658211    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:21.658193815 +0000 UTC m=+268.987633727 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:17.845784 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:17.845758    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7lj4_a599a220-6fa7-4d9e-be51-52258ab5f6c0/node-ca/0.log"
Jul 16 21:51:18.246688 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:18.246659    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prpt_9fe2c42b-98c5-491d-b75d-a817769e7c85/serve-healthcheck-canary/0.log"
Jul 16 21:51:18.992814 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:18.992721    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-r2zth" event={"ID":"12a42f6a-35a5-4790-a03a-0a5334058a1d","Type":"ContainerStarted","Data":"1c82f9c534b5df09df936a5d55b274cd6b6beddff6b8be496842a9fd5e226695"}
Jul 16 21:51:19.009116 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:19.009071    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-r2zth" podStartSLOduration=1.217107838 podStartE2EDuration="3.009054715s" totalImagesPullingTime="1.791946877s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:19.008627901 +0000 UTC m=+266.338067805" watchObservedRunningTime="2026-07-16 21:51:19.009054715 +0000 UTC m=+266.338494596"
Jul 16 21:51:21.688058 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:21.688028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:21.688058 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:21.688062    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:21.688527 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:21.688165    2581 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:21.688527 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:21.688179    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:21.688527 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:21.688218    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:29.688204953 +0000 UTC m=+277.017644835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:21.688527 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:21.688260    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:29.688224204 +0000 UTC m=+277.017664085 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:25.015531 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:25.015498    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:25.015955 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:25.015625    2581 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:25.015955 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:25.015677    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert podName:f7341c14-3ac0-4f15-8cd1-1f88bdf1717e nodeName:}" failed. No retries permitted until 2026-07-16 21:51:41.015664274 +0000 UTC m=+288.345104155 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-645r4" (UID: "f7341c14-3ac0-4f15-8cd1-1f88bdf1717e") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:29.753631 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:29.753592    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:29.754113 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:29.753641    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:29.754113 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:29.753757    2581 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:29.754113 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:29.753846    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls podName:6bc30111-495d-430a-9792-1a3021fd6c78 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:45.753829318 +0000 UTC m=+293.083269201 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2455k" (UID: "6bc30111-495d-430a-9792-1a3021fd6c78") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:29.754322 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:29.754224    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/6bc30111-495d-430a-9792-1a3021fd6c78-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:35.701711 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:35.701661    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-fx9mf" podUID="12cd7b07-a130-4366-b420-03897d2ee326"
Jul 16 21:51:36.043896 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:36.043819    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:38.959803 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.959772    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5cbdf84548-xlnd9"]
Jul 16 21:51:38.963023 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.963007    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:38.965890 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.965868    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:51:38.965992 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.965870    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j8r94\""
Jul 16 21:51:38.965992 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.965943    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:51:38.966324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.966311    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:51:38.971083 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.971067    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:51:38.983953 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:38.983928    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cbdf84548-xlnd9"]
Jul 16 21:51:39.025168 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025138    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-ca-trust-extracted\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025301 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025184    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-tls\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025301 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-certificates\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025301 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025263    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:39.025301 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025292    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28pmp\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-kube-api-access-28pmp\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025499 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025312    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-installation-pull-secrets\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025499 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025371    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-trusted-ca\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025499 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025403    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-image-registry-private-configuration\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.025499 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.025434    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-bound-sa-token\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.027765 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.027744    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12cd7b07-a130-4366-b420-03897d2ee326-metrics-tls\") pod \"dns-default-fx9mf\" (UID: \"12cd7b07-a130-4366-b420-03897d2ee326\") " pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:39.047217 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.047194    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-275zl\""
Jul 16 21:51:39.054991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.054976    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:39.126476 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28pmp\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-kube-api-access-28pmp\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126639 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126492    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-installation-pull-secrets\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126639 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126530    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-trusted-ca\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126639 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126557    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-image-registry-private-configuration\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126639 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126583    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-bound-sa-token\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126846 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126684    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-ca-trust-extracted\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.126934 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126912    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-tls\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.127003 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.126950    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-certificates\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.127381 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.127324    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-ca-trust-extracted\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.127720 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.127699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-trusted-ca\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.128048 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.128022    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-certificates\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.129379 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.129359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-installation-pull-secrets\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.129458 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.129383    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-image-registry-private-configuration\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.129718 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.129703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-registry-tls\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.142911 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.142881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-bound-sa-token\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.144906 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.144882    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28pmp\" (UniqueName: \"kubernetes.io/projected/a50e695f-fbcf-4c8e-9dfa-a56259f9abe6-kube-api-access-28pmp\") pod \"image-registry-5cbdf84548-xlnd9\" (UID: \"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6\") " pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.192833 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.192813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-fx9mf"]
Jul 16 21:51:39.196521 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:39.196493    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cd7b07_a130_4366_b420_03897d2ee326.slice/crio-6a84225b96ebd2d27c110962d62e75510bdd973b2f6a7159a9cc0b00d29b950f WatchSource:0}: Error finding container 6a84225b96ebd2d27c110962d62e75510bdd973b2f6a7159a9cc0b00d29b950f: Status 404 returned error can't find the container with id 6a84225b96ebd2d27c110962d62e75510bdd973b2f6a7159a9cc0b00d29b950f
Jul 16 21:51:39.273282 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.273205    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:39.455441 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:39.455411    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5cbdf84548-xlnd9"]
Jul 16 21:51:39.458659 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:39.458633    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda50e695f_fbcf_4c8e_9dfa_a56259f9abe6.slice/crio-e88810a305bb0c4d2b59aae5a0f8c3524d27e52271092a5880dad1bc1db3011c WatchSource:0}: Error finding container e88810a305bb0c4d2b59aae5a0f8c3524d27e52271092a5880dad1bc1db3011c: Status 404 returned error can't find the container with id e88810a305bb0c4d2b59aae5a0f8c3524d27e52271092a5880dad1bc1db3011c
Jul 16 21:51:40.058241 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:40.058189    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx9mf" event={"ID":"12cd7b07-a130-4366-b420-03897d2ee326","Type":"ContainerStarted","Data":"6a84225b96ebd2d27c110962d62e75510bdd973b2f6a7159a9cc0b00d29b950f"}
Jul 16 21:51:40.059845 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:40.059816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9" event={"ID":"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6","Type":"ContainerStarted","Data":"53d190eab8dec0b079c5db76f7e30b14a5ca6323140704d6afdb2a15ea47b179"}
Jul 16 21:51:40.059960 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:40.059863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9" event={"ID":"a50e695f-fbcf-4c8e-9dfa-a56259f9abe6","Type":"ContainerStarted","Data":"e88810a305bb0c4d2b59aae5a0f8c3524d27e52271092a5880dad1bc1db3011c"}
Jul 16 21:51:40.059960 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:40.059901    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:51:40.088061 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:40.088014    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9" podStartSLOduration=2.087993607 podStartE2EDuration="2.087993607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:40.087580648 +0000 UTC m=+287.417020552" watchObservedRunningTime="2026-07-16 21:51:40.087993607 +0000 UTC m=+287.417433511"
Jul 16 21:51:41.042930 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:41.042895    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:41.046506 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:41.046456    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f7341c14-3ac0-4f15-8cd1-1f88bdf1717e-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-645r4\" (UID: \"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:41.065108 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:41.064994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx9mf" event={"ID":"12cd7b07-a130-4366-b420-03897d2ee326","Type":"ContainerStarted","Data":"01833e29feb000b2ca18e085db06f564ab1724a332b3e8a3a44a4d2802e286ba"}
Jul 16 21:51:41.339184 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:41.339156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"
Jul 16 21:51:41.471710 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:41.471678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-645r4"]
Jul 16 21:51:41.474600 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:41.474577    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7341c14_3ac0_4f15_8cd1_1f88bdf1717e.slice/crio-a752f40419c4d1d59e4dc7248ae14b0ada38b80c82c7c965e24fa59fe4d56b55 WatchSource:0}: Error finding container a752f40419c4d1d59e4dc7248ae14b0ada38b80c82c7c965e24fa59fe4d56b55: Status 404 returned error can't find the container with id a752f40419c4d1d59e4dc7248ae14b0ada38b80c82c7c965e24fa59fe4d56b55
Jul 16 21:51:42.070903 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:42.070867    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-fx9mf" event={"ID":"12cd7b07-a130-4366-b420-03897d2ee326","Type":"ContainerStarted","Data":"d701f9de4fb40a0f52f825fa7b6bb8ab5ba1790d6b3e5d5737f294a7fc05eb0e"}
Jul 16 21:51:42.071406 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:42.070975    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:42.072247 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:42.072206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4" event={"ID":"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e","Type":"ContainerStarted","Data":"a752f40419c4d1d59e4dc7248ae14b0ada38b80c82c7c965e24fa59fe4d56b55"}
Jul 16 21:51:42.090411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:42.090367    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-fx9mf" podStartSLOduration=252.347580881 podStartE2EDuration="4m14.090352024s" totalImagesPullingTime="1.742771143s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:47:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:42.089002112 +0000 UTC m=+289.418442014" watchObservedRunningTime="2026-07-16 21:51:42.090352024 +0000 UTC m=+289.419791926"
Jul 16 21:51:43.076480 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:43.076440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4" event={"ID":"f7341c14-3ac0-4f15-8cd1-1f88bdf1717e","Type":"ContainerStarted","Data":"1e70444946739afad32bf3beb041203bd6834cecae628783bf76cab0f1d1779c"}
Jul 16 21:51:45.778368 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:45.778335    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:45.780730 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:45.780711    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/6bc30111-495d-430a-9792-1a3021fd6c78-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2455k\" (UID: \"6bc30111-495d-430a-9792-1a3021fd6c78\") " pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:45.979572 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:45.979514    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2455k"
Jul 16 21:51:46.127638 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:46.127595    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bc30111_495d_430a_9792_1a3021fd6c78.slice/crio-15dec134ed56966a72e45c4a702fc88b0229a77566f9145192799aaf80fda682 WatchSource:0}: Error finding container 15dec134ed56966a72e45c4a702fc88b0229a77566f9145192799aaf80fda682: Status 404 returned error can't find the container with id 15dec134ed56966a72e45c4a702fc88b0229a77566f9145192799aaf80fda682
Jul 16 21:51:46.128899 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:46.128855    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-645r4" podStartSLOduration=36.091357813 podStartE2EDuration="37.128839621s" totalImagesPullingTime="1.037481808s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:43.095912144 +0000 UTC m=+290.425352051" watchObservedRunningTime="2026-07-16 21:51:46.128839621 +0000 UTC m=+293.458279555"
Jul 16 21:51:46.129783 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:46.129763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2455k"]
Jul 16 21:51:46.158384 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:46.158344    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-sxzbc" podUID="cebc9e23-3745-4d0f-b03f-7b59a152c0e4"
Jul 16 21:51:47.090856 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:47.090817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2455k" event={"ID":"6bc30111-495d-430a-9792-1a3021fd6c78","Type":"ContainerStarted","Data":"15dec134ed56966a72e45c4a702fc88b0229a77566f9145192799aaf80fda682"}
Jul 16 21:51:51.104101 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:51.104063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2455k" event={"ID":"6bc30111-495d-430a-9792-1a3021fd6c78","Type":"ContainerStarted","Data":"5a28a64e5eefe0ef47813b1dc0ce4dbb8bc4be03fa43dfb991fe7a1ebda6698c"}
Jul 16 21:51:52.079243 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:52.079194    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-fx9mf"
Jul 16 21:51:54.133322 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.133284    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:51:54.145458 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.145431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.154503 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.154320    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:51:54.154503 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.154410    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:51:54.155698 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.155514    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:51:54.156633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.156460    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:51:54.156633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.156471    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dbx5n\""
Jul 16 21:51:54.156633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.156550    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:51:54.156841 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.156732    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:51:54.157171 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.157154    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:51:54.159086 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.159065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:51:54.160622 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.160605    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:51:54.246639 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246603    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246695    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246739    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246755    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tss2l\" (UniqueName: \"kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.246825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.246822    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.297009 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.296986    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:51:54.297334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.297319    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:51:54.300655 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.300637    2581 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:51:54.347648 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.347624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.347744 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.347657    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.347744 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.347694    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.347744 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.347717    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348038    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348080    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tss2l\" (UniqueName: \"kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348108    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348484 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348753 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348676    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348753 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348701    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.348753 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.348750    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.350524 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.350505    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.350907 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.350889    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.364336 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.364314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tss2l\" (UniqueName: \"kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l\") pod \"console-5758fc956c-p5f4r\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") " pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.401177 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.401157    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:51:54.456041 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.456005    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:51:54.612302 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.612270    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:51:54.617491 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:54.617469    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03876f85_f7a4_4458_a30e_f7fd745725e9.slice/crio-d91fb8cc7fd5c35629e385591b44e252e2fa622ee928bddebfdd171b24048c4c WatchSource:0}: Error finding container d91fb8cc7fd5c35629e385591b44e252e2fa622ee928bddebfdd171b24048c4c: Status 404 returned error can't find the container with id d91fb8cc7fd5c35629e385591b44e252e2fa622ee928bddebfdd171b24048c4c
Jul 16 21:51:54.909252 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.909159    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qppn2"]
Jul 16 21:51:54.913068 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.912412    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916361    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916416    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916423    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916546    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916745    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916831    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:51:54.917315 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.916857    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-vr4nd\""
Jul 16 21:51:54.926857 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.926834    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qppn2"]
Jul 16 21:51:54.932340 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.932312    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-vjq95"]
Jul 16 21:51:54.935517 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.935496    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:54.938439 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.938415    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:54.938641 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.938627    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:54.938950 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.938937    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-l4dh8\""
Jul 16 21:51:54.939120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.939106    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:54.952993 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.952965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.953120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.953009    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.953120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.953052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4675264c-f22e-444a-b441-85f30adc3ef7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.953120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.953106    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.953368 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.953133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:54.953368 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:54.953162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk6tc\" (UniqueName: \"kubernetes.io/projected/4675264c-f22e-444a-b441-85f30adc3ef7-kube-api-access-lk6tc\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054035 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.053987    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054193 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054036    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-wtmp\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054193 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054087    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8dl\" (UniqueName: \"kubernetes.io/projected/a558964d-b2aa-4cce-9733-aa8162b70c38-kube-api-access-xv8dl\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054193 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054117    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4675264c-f22e-444a-b441-85f30adc3ef7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054193 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054147    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-metrics-client-ca\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054201    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054281    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-sys\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk6tc\" (UniqueName: \"kubernetes.io/projected/4675264c-f22e-444a-b441-85f30adc3ef7-kube-api-access-lk6tc\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054423 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054391    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-root\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054451    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-textfile\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.054697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054477    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.054697 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.054505    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.055103 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:55.055081    2581 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 16 21:51:55.055497 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.055471    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/4675264c-f22e-444a-b441-85f30adc3ef7-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.055609 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:55.055517    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls podName:4675264c-f22e-444a-b441-85f30adc3ef7 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:55.55549295 +0000 UTC m=+302.884932837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-qppn2" (UID: "4675264c-f22e-444a-b441-85f30adc3ef7") : secret "kube-state-metrics-tls" not found
Jul 16 21:51:55.056104 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.056074    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.056681 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.056405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/4675264c-f22e-444a-b441-85f30adc3ef7-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.058007 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.057971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.071377 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.071329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk6tc\" (UniqueName: \"kubernetes.io/projected/4675264c-f22e-444a-b441-85f30adc3ef7-kube-api-access-lk6tc\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.123046 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.122964    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2455k" event={"ID":"6bc30111-495d-430a-9792-1a3021fd6c78","Type":"ContainerStarted","Data":"48d72c344c82c87f3e98b8704d9338f68e4cbfde9a2ceb85e0d49eb2d1ce641f"}
Jul 16 21:51:55.124800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.124766    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5758fc956c-p5f4r" event={"ID":"03876f85-f7a4-4458-a30e-f7fd745725e9","Type":"ContainerStarted","Data":"d91fb8cc7fd5c35629e385591b44e252e2fa622ee928bddebfdd171b24048c4c"}
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155638    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-root\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-textfile\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155711    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155738    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-wtmp\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-root\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.155800 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155779    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8dl\" (UniqueName: \"kubernetes.io/projected/a558964d-b2aa-4cce-9733-aa8162b70c38-kube-api-access-xv8dl\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155825    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-metrics-client-ca\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155938    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-sys\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.155977    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:55.156098    2581 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:51:55.156150    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls podName:a558964d-b2aa-4cce-9733-aa8162b70c38 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:55.656131641 +0000 UTC m=+302.985571527 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls") pod "node-exporter-vjq95" (UID: "a558964d-b2aa-4cce-9733-aa8162b70c38") : secret "node-exporter-tls" not found
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.156381    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-sys\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.156438    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-textfile\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156579 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.156574    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-wtmp\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.156935 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.156767    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-accelerators-collector-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.157017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.156997    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a558964d-b2aa-4cce-9733-aa8162b70c38-metrics-client-ca\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.159942 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.159864    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.165428 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.165403    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8dl\" (UniqueName: \"kubernetes.io/projected/a558964d-b2aa-4cce-9733-aa8162b70c38-kube-api-access-xv8dl\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.559844 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.559765    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.562934 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.562906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/4675264c-f22e-444a-b441-85f30adc3ef7-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-qppn2\" (UID: \"4675264c-f22e-444a-b441-85f30adc3ef7\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.660395 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.660311    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.665196 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.665166    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/a558964d-b2aa-4cce-9733-aa8162b70c38-node-exporter-tls\") pod \"node-exporter-vjq95\" (UID: \"a558964d-b2aa-4cce-9733-aa8162b70c38\") " pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.827924 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.827848    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2"
Jul 16 21:51:55.853245 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.852767    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-vjq95"
Jul 16 21:51:55.864983 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:55.864956    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda558964d_b2aa_4cce_9733_aa8162b70c38.slice/crio-466264aca149be1e454d5d3fa156da16971bf343d0df15f59251ba60d61aeebe WatchSource:0}: Error finding container 466264aca149be1e454d5d3fa156da16971bf343d0df15f59251ba60d61aeebe: Status 404 returned error can't find the container with id 466264aca149be1e454d5d3fa156da16971bf343d0df15f59251ba60d61aeebe
Jul 16 21:51:55.985934 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:55.985786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-qppn2"]
Jul 16 21:51:55.988690 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:51:55.988661    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4675264c_f22e_444a_b441_85f30adc3ef7.slice/crio-cc9afdd2b483e9d1397729ac81be4413ac4d2bc83fc2fa34ec240e3ed3ce59cc WatchSource:0}: Error finding container cc9afdd2b483e9d1397729ac81be4413ac4d2bc83fc2fa34ec240e3ed3ce59cc: Status 404 returned error can't find the container with id cc9afdd2b483e9d1397729ac81be4413ac4d2bc83fc2fa34ec240e3ed3ce59cc
Jul 16 21:51:56.130378 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:56.130339    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjq95" event={"ID":"a558964d-b2aa-4cce-9733-aa8162b70c38","Type":"ContainerStarted","Data":"466264aca149be1e454d5d3fa156da16971bf343d0df15f59251ba60d61aeebe"}
Jul 16 21:51:56.132280 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:51:56.132247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2" event={"ID":"4675264c-f22e-444a-b441-85f30adc3ef7","Type":"ContainerStarted","Data":"cc9afdd2b483e9d1397729ac81be4413ac4d2bc83fc2fa34ec240e3ed3ce59cc"}
Jul 16 21:52:00.148117 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.148085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5758fc956c-p5f4r" event={"ID":"03876f85-f7a4-4458-a30e-f7fd745725e9","Type":"ContainerStarted","Data":"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"}
Jul 16 21:52:00.150492 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.150464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2" event={"ID":"4675264c-f22e-444a-b441-85f30adc3ef7","Type":"ContainerStarted","Data":"766da3bb9d2b2b7600c5ba73e2ea3aeb66167edd848519c4b97ac2704b4adcf4"}
Jul 16 21:52:00.150600 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.150522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2" event={"ID":"4675264c-f22e-444a-b441-85f30adc3ef7","Type":"ContainerStarted","Data":"1a5cc11d47f63a31441b5c6e038db555e485fb674ecb4b3c74bf99d7f113d9aa"}
Jul 16 21:52:00.150600 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.150545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2" event={"ID":"4675264c-f22e-444a-b441-85f30adc3ef7","Type":"ContainerStarted","Data":"0d7ee86e45b651891f49050c4f4ca7337a4e16f6fc5f4b5b20b2d0c90593b6b6"}
Jul 16 21:52:00.151908 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.151888    2581 generic.go:356] "Generic (PLEG): container finished" podID="a558964d-b2aa-4cce-9733-aa8162b70c38" containerID="d321cc6b8afc777df263c7cca190dcf52c91a88af1538027869f0ee76cecca78" exitCode=0
Jul 16 21:52:00.151991 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.151968    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjq95" event={"ID":"a558964d-b2aa-4cce-9733-aa8162b70c38","Type":"ContainerDied","Data":"d321cc6b8afc777df263c7cca190dcf52c91a88af1538027869f0ee76cecca78"}
Jul 16 21:52:00.154076 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.154053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2455k" event={"ID":"6bc30111-495d-430a-9792-1a3021fd6c78","Type":"ContainerStarted","Data":"0c2b68c10d2f83e8aa7234bcbf9a764e00ee97a447684dac9b24b5e3630f7da3"}
Jul 16 21:52:00.185407 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.185361    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5758fc956c-p5f4r" podStartSLOduration=1.023663737 podStartE2EDuration="6.185344388s" totalImagesPullingTime="5.161680651s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:00.183053239 +0000 UTC m=+307.512493145" watchObservedRunningTime="2026-07-16 21:52:00.185344388 +0000 UTC m=+307.514784342"
Jul 16 21:52:00.290867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.290813    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2455k" podStartSLOduration=33.759434867 podStartE2EDuration="47.29079524s" totalImagesPullingTime="13.531360373s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:00.229995241 +0000 UTC m=+307.559435169" watchObservedRunningTime="2026-07-16 21:52:00.29079524 +0000 UTC m=+307.620235142"
Jul 16 21:52:00.291198 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:00.291170    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-qppn2" podStartSLOduration=2.500764686 podStartE2EDuration="6.291158757s" totalImagesPullingTime="3.790394071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:00.288206095 +0000 UTC m=+307.617646003" watchObservedRunningTime="2026-07-16 21:52:00.291158757 +0000 UTC m=+307.620598696"
Jul 16 21:52:01.070283 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:01.070204    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5cbdf84548-xlnd9"
Jul 16 21:52:01.154628 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:01.154591    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:52:01.160057 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:01.160024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjq95" event={"ID":"a558964d-b2aa-4cce-9733-aa8162b70c38","Type":"ContainerStarted","Data":"6ea886f06bbcb69bef09af1728df051da37137636487cb53604eb867af1ace50"}
Jul 16 21:52:01.160199 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:01.160075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-vjq95" event={"ID":"a558964d-b2aa-4cce-9733-aa8162b70c38","Type":"ContainerStarted","Data":"be9ab4fd14e1674667bff9e718083a5ed8e69d34847bd7b40849fbd801a6686c"}
Jul 16 21:52:01.195510 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:01.195462    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-vjq95" podStartSLOduration=3.128893163 podStartE2EDuration="7.195446691s" totalImagesPullingTime="3.96553502s" totalInitContainerRuntime="101.018508ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:01.194072634 +0000 UTC m=+308.523512535" watchObservedRunningTime="2026-07-16 21:52:01.195446691 +0000 UTC m=+308.524886616"
Jul 16 21:52:02.678010 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:02.677977    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:52:04.456778 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:04.456742    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:52:27.698260 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:27.698165    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5758fc956c-p5f4r" podUID="03876f85-f7a4-4458-a30e-f7fd745725e9" containerName="console" containerID="cri-o://b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840" gracePeriod=15
Jul 16 21:52:27.966538 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:27.966482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:52:28.129311 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tss2l\" (UniqueName: \"kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129477 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129477 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129382    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129477 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129477 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129445    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129647 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129479    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129647 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert\") pod \"03876f85-f7a4-4458-a30e-f7fd745725e9\" (UID: \"03876f85-f7a4-4458-a30e-f7fd745725e9\") "
Jul 16 21:52:28.129885 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129811    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config" (OuterVolumeSpecName: "console-config") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:28.129974 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129929    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:28.129974 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.129940    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca" (OuterVolumeSpecName: "service-ca") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:28.130279 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.130222    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:28.131707 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.131681    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l" (OuterVolumeSpecName: "kube-api-access-tss2l") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "kube-api-access-tss2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:28.131707 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.131697    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:28.131823 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.131754    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "03876f85-f7a4-4458-a30e-f7fd745725e9" (UID: "03876f85-f7a4-4458-a30e-f7fd745725e9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230785    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-console-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230814    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230825    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-trusted-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230836    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/03876f85-f7a4-4458-a30e-f7fd745725e9-console-oauth-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230845    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230854    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/03876f85-f7a4-4458-a30e-f7fd745725e9-oauth-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.230869 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.230862    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tss2l\" (UniqueName: \"kubernetes.io/projected/03876f85-f7a4-4458-a30e-f7fd745725e9-kube-api-access-tss2l\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:52:28.248728 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.248698    2581 generic.go:356] "Generic (PLEG): container finished" podID="03876f85-f7a4-4458-a30e-f7fd745725e9" containerID="b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840" exitCode=0
Jul 16 21:52:28.248848 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.248738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5758fc956c-p5f4r" event={"ID":"03876f85-f7a4-4458-a30e-f7fd745725e9","Type":"ContainerDied","Data":"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"}
Jul 16 21:52:28.248848 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.248748    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5758fc956c-p5f4r"
Jul 16 21:52:28.248848 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.248790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5758fc956c-p5f4r" event={"ID":"03876f85-f7a4-4458-a30e-f7fd745725e9","Type":"ContainerDied","Data":"d91fb8cc7fd5c35629e385591b44e252e2fa622ee928bddebfdd171b24048c4c"}
Jul 16 21:52:28.248848 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.248818    2581 scope.go:117] "RemoveContainer" containerID="b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"
Jul 16 21:52:28.257161 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.257143    2581 scope.go:117] "RemoveContainer" containerID="b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"
Jul 16 21:52:28.257466 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:52:28.257447    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840\": container with ID starting with b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840 not found: ID does not exist" containerID="b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"
Jul 16 21:52:28.257527 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.257474    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840"} err="failed to get container status \"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840\": rpc error: code = NotFound desc = could not find container \"b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840\": container with ID starting with b7071780dc575c032964b79878da6c3abed6e7d2117e8a297851b6e4495b7840 not found: ID does not exist"
Jul 16 21:52:28.269333 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.269312    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:52:28.275386 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:28.275353    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5758fc956c-p5f4r"]
Jul 16 21:52:29.159027 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:29.158982    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="03876f85-f7a4-4458-a30e-f7fd745725e9" path="/var/lib/kubelet/pods/03876f85-f7a4-4458-a30e-f7fd745725e9/volumes"
Jul 16 21:52:29.275021 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:29.274990    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prpt_9fe2c42b-98c5-491d-b75d-a817769e7c85/serve-healthcheck-canary/0.log"
Jul 16 21:52:39.405065 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:39.405010    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" podUID="1799669f-da56-435c-a929-ac4b1273400b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 21:52:49.405353 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:49.405270    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" podUID="1799669f-da56-435c-a929-ac4b1273400b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 21:52:59.405117 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:59.405076    2581 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" podUID="1799669f-da56-435c-a929-ac4b1273400b" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 16 21:52:59.405611 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:59.405151    2581 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk"
Jul 16 21:52:59.405655 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:59.405641    2581 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"b5b47f3465c644dd15aafdad3d3423937b5a728c9cf29df04a3321b6266292d6"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 16 21:52:59.405693 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:52:59.405677    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" podUID="1799669f-da56-435c-a929-ac4b1273400b" containerName="service-proxy" containerID="cri-o://b5b47f3465c644dd15aafdad3d3423937b5a728c9cf29df04a3321b6266292d6" gracePeriod=30
Jul 16 21:53:00.357528 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:00.357496    2581 generic.go:356] "Generic (PLEG): container finished" podID="1799669f-da56-435c-a929-ac4b1273400b" containerID="b5b47f3465c644dd15aafdad3d3423937b5a728c9cf29df04a3321b6266292d6" exitCode=2
Jul 16 21:53:00.357708 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:00.357545    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerDied","Data":"b5b47f3465c644dd15aafdad3d3423937b5a728c9cf29df04a3321b6266292d6"}
Jul 16 21:53:00.357708 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:00.357579    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-55db467cc7-zr6mk" event={"ID":"1799669f-da56-435c-a929-ac4b1273400b","Type":"ContainerStarted","Data":"b47dce25fe761d58589c4aec5ada14d096d3b98d1605499fe3cee0b15556bd11"}
Jul 16 21:53:07.027903 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.027863    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:53:07.030311 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.030290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/cebc9e23-3745-4d0f-b03f-7b59a152c0e4-metrics-certs\") pod \"network-metrics-daemon-sxzbc\" (UID: \"cebc9e23-3745-4d0f-b03f-7b59a152c0e4\") " pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:53:07.158342 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.158313    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-gkkkc\""
Jul 16 21:53:07.166678 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.166655    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-sxzbc"
Jul 16 21:53:07.294625 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.294534    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-sxzbc"]
Jul 16 21:53:07.298278 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:53:07.298251    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcebc9e23_3745_4d0f_b03f_7b59a152c0e4.slice/crio-ae8d18a148f47043c13cb8a08b5dce4b4648c4ebbc694a032c78542e7a4a81b0 WatchSource:0}: Error finding container ae8d18a148f47043c13cb8a08b5dce4b4648c4ebbc694a032c78542e7a4a81b0: Status 404 returned error can't find the container with id ae8d18a148f47043c13cb8a08b5dce4b4648c4ebbc694a032c78542e7a4a81b0
Jul 16 21:53:07.379763 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:07.379720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxzbc" event={"ID":"cebc9e23-3745-4d0f-b03f-7b59a152c0e4","Type":"ContainerStarted","Data":"ae8d18a148f47043c13cb8a08b5dce4b4648c4ebbc694a032c78542e7a4a81b0"}
Jul 16 21:53:09.387538 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:09.387497    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxzbc" event={"ID":"cebc9e23-3745-4d0f-b03f-7b59a152c0e4","Type":"ContainerStarted","Data":"be948c60d23ce1b9ea790d101def33305c67996cd6f4b75a8b89e2e62e896fc1"}
Jul 16 21:53:09.387906 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:09.387557    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-sxzbc" event={"ID":"cebc9e23-3745-4d0f-b03f-7b59a152c0e4","Type":"ContainerStarted","Data":"0085b5e28636a460ad3be842eaddbb82cc132eb257820b72c77d6b5789a60a5d"}
Jul 16 21:53:09.407351 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:09.407303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-sxzbc" podStartSLOduration=375.398402804 podStartE2EDuration="6m16.407281366s" totalImagesPullingTime="1.008878562s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:09.405653529 +0000 UTC m=+376.735093443" watchObservedRunningTime="2026-07-16 21:53:09.407281366 +0000 UTC m=+376.736721273"
Jul 16 21:53:31.033847 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.033812    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:53:31.034273 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.034096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="03876f85-f7a4-4458-a30e-f7fd745725e9" containerName="console"
Jul 16 21:53:31.034273 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.034106    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="03876f85-f7a4-4458-a30e-f7fd745725e9" containerName="console"
Jul 16 21:53:31.034273 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.034150    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="03876f85-f7a4-4458-a30e-f7fd745725e9" containerName="console"
Jul 16 21:53:31.037670 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.037651    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.042688 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.042667    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:53:31.042853 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.042827    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:53:31.043159 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.043139    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:53:31.043270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.043202    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:53:31.043270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.043169    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-dbx5n\""
Jul 16 21:53:31.044099 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.043780    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:53:31.044461 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.044436    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:53:31.044828 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.044813    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:53:31.049100 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.049081    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:53:31.055673 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.055654    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:53:31.196194 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196161    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196194 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196193    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kw68\" (UniqueName: \"kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196217    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196276    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196321    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.196411 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.196355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297638 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297537    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297638 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297638 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297621    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297919 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297919 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297712    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297919 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297733    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kw68\" (UniqueName: \"kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.297919 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.297758    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.298432 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.298404    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.298619 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.298439    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.298619 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.298561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.298807 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.298792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.300277 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.300255    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.300382 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.300290    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.318783 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.318748    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kw68\" (UniqueName: \"kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68\") pod \"console-75f98f54f6-qzwfn\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") " pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.346826 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.346787    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:31.506189 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:31.506160    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:53:31.509912 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:53:31.509885    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66c204a4_49fd_4589_ae29_b6a480b66dec.slice/crio-26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b WatchSource:0}: Error finding container 26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b: Status 404 returned error can't find the container with id 26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b
Jul 16 21:53:32.472171 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:32.472123    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f98f54f6-qzwfn" event={"ID":"66c204a4-49fd-4589-ae29-b6a480b66dec","Type":"ContainerStarted","Data":"0ad46b0c9bdef8d592c5c0e58b68e78abc65783e0e1592eb20c2b3b3b90fc8c8"}
Jul 16 21:53:32.472171 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:32.472178    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f98f54f6-qzwfn" event={"ID":"66c204a4-49fd-4589-ae29-b6a480b66dec","Type":"ContainerStarted","Data":"26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b"}
Jul 16 21:53:41.347426 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:41.347371    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:41.347905 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:41.347448    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:41.352261 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:41.352238    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:53:41.377572 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:41.377522    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-75f98f54f6-qzwfn" podStartSLOduration=10.377506125 podStartE2EDuration="10.377506125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:32.517343979 +0000 UTC m=+399.846783882" watchObservedRunningTime="2026-07-16 21:53:41.377506125 +0000 UTC m=+408.706946030"
Jul 16 21:53:41.505682 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:53:41.505660    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:54:55.120474 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.120442    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ktskh"]
Jul 16 21:54:55.123427 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.123409    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.140642 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.140622    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-67zrp\""
Jul 16 21:54:55.155023 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.154999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 21:54:55.159658 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.159638    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ktskh"]
Jul 16 21:54:55.165448 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.165430    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 21:54:55.212707 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.212673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfkx5\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-kube-api-access-tfkx5\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.212876 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.212713    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-bound-sa-token\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.313086 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.313029    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-bound-sa-token\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.313297 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.313136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfkx5\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-kube-api-access-tfkx5\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.336002 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.335971    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-bound-sa-token\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.336130 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.336035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfkx5\" (UniqueName: \"kubernetes.io/projected/bdf08b47-5d0b-4e24-9749-f79a5d35bcbd-kube-api-access-tfkx5\") pod \"cert-manager-55b4d954d5-ktskh\" (UID: \"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd\") " pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.432482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.432381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-ktskh"
Jul 16 21:54:55.564398 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.564368    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-ktskh"]
Jul 16 21:54:55.567271 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:54:55.567223    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbdf08b47_5d0b_4e24_9749_f79a5d35bcbd.slice/crio-28239be39f0a353482e4b1df6cb3b330e7bf07e8c20feba0209932a4f37f3601 WatchSource:0}: Error finding container 28239be39f0a353482e4b1df6cb3b330e7bf07e8c20feba0209932a4f37f3601: Status 404 returned error can't find the container with id 28239be39f0a353482e4b1df6cb3b330e7bf07e8c20feba0209932a4f37f3601
Jul 16 21:54:55.743655 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:55.743568    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ktskh" event={"ID":"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd","Type":"ContainerStarted","Data":"28239be39f0a353482e4b1df6cb3b330e7bf07e8c20feba0209932a4f37f3601"}
Jul 16 21:54:58.757425 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:58.757387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-ktskh" event={"ID":"bdf08b47-5d0b-4e24-9749-f79a5d35bcbd","Type":"ContainerStarted","Data":"8720ffc8a6de17f1670cefbaad2ab5de6352c370ed5eebf0508613bbb78f1902"}
Jul 16 21:54:58.775376 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:54:58.775331    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-ktskh" podStartSLOduration=1.044534323 podStartE2EDuration="3.775314648s" totalImagesPullingTime="2.730780325s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:58.772891379 +0000 UTC m=+486.102331280" watchObservedRunningTime="2026-07-16 21:54:58.775314648 +0000 UTC m=+486.104754551"
Jul 16 21:55:21.639446 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.639409    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"]
Jul 16 21:55:21.642604 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.642585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.648120 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.648099    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 21:55:21.648406 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.648386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-q5qbl\""
Jul 16 21:55:21.648482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.648395    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 21:55:21.648482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.648422    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 21:55:21.649069 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.648686    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 21:55:21.679124 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.679097    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"]
Jul 16 21:55:21.809492 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.809439    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7l7h\" (UniqueName: \"kubernetes.io/projected/a852da53-3657-47a1-b8f0-78f8dfd0985a-kube-api-access-z7l7h\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.809492 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.809491    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.809722 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.809548    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.809722 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.809583    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.809722 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.809622    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910213 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910126    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910213 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910266    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7l7h\" (UniqueName: \"kubernetes.io/projected/a852da53-3657-47a1-b8f0-78f8dfd0985a-kube-api-access-z7l7h\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910291    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910478 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910329    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.910723 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.910703    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-manifests\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.912658 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.912628    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a852da53-3657-47a1-b8f0-78f8dfd0985a-tmp\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.912867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.912852    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-webhook-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.912918 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.912884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/a852da53-3657-47a1-b8f0-78f8dfd0985a-apiservice-cert\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.926515 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.926491    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7l7h\" (UniqueName: \"kubernetes.io/projected/a852da53-3657-47a1-b8f0-78f8dfd0985a-kube-api-access-z7l7h\") pod \"opendatahub-operator-controller-manager-df6666d-pzzpd\" (UID: \"a852da53-3657-47a1-b8f0-78f8dfd0985a\") " pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:21.952385 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:21.952357    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:22.122497 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:22.122466    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"]
Jul 16 21:55:22.126377 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:55:22.126347    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda852da53_3657_47a1_b8f0_78f8dfd0985a.slice/crio-2a8b4bfa6eadcb0ce49aea5b7b9397b32cc6196d09a012039c42d2b03450a524 WatchSource:0}: Error finding container 2a8b4bfa6eadcb0ce49aea5b7b9397b32cc6196d09a012039c42d2b03450a524: Status 404 returned error can't find the container with id 2a8b4bfa6eadcb0ce49aea5b7b9397b32cc6196d09a012039c42d2b03450a524
Jul 16 21:55:22.840766 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:22.840719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd" event={"ID":"a852da53-3657-47a1-b8f0-78f8dfd0985a","Type":"ContainerStarted","Data":"2a8b4bfa6eadcb0ce49aea5b7b9397b32cc6196d09a012039c42d2b03450a524"}
Jul 16 21:55:25.854512 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:25.854475    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd" event={"ID":"a852da53-3657-47a1-b8f0-78f8dfd0985a","Type":"ContainerStarted","Data":"3886f7ba62351e107552f8e76e10c49f840538df8ec5c0f9d46bdb2b20c956f8"}
Jul 16 21:55:26.859367 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:26.859335    2581 generic.go:356] "Generic (PLEG): container finished" podID="a852da53-3657-47a1-b8f0-78f8dfd0985a" containerID="3886f7ba62351e107552f8e76e10c49f840538df8ec5c0f9d46bdb2b20c956f8" exitCode=0
Jul 16 21:55:26.859867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:26.859430    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd" event={"ID":"a852da53-3657-47a1-b8f0-78f8dfd0985a","Type":"ContainerDied","Data":"3886f7ba62351e107552f8e76e10c49f840538df8ec5c0f9d46bdb2b20c956f8"}
Jul 16 21:55:27.865039 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:27.865000    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd" event={"ID":"a852da53-3657-47a1-b8f0-78f8dfd0985a","Type":"ContainerStarted","Data":"6934a2950016fd18e985a82415f955c3228940678062043431563101f0f7347a"}
Jul 16 21:55:27.865491 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:27.865150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:27.938698 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:27.938638    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd" podStartSLOduration=2.618962125 podStartE2EDuration="6.938618849s" totalImagesPullingTime="3.298181614s" totalInitContainerRuntime="1.02147511s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:21 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:27.934988264 +0000 UTC m=+515.264428177" watchObservedRunningTime="2026-07-16 21:55:27.938618849 +0000 UTC m=+515.268058753"
Jul 16 21:55:38.870994 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:38.870956    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-df6666d-pzzpd"
Jul 16 21:55:43.047512 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.047475    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"]
Jul 16 21:55:43.055177 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.055154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.068169 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.068146    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:55:43.068335 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.068317    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xtcl2\" (UniqueName: \"kubernetes.io/projected/30709861-ba73-4a0e-8306-5f3e1b991dee-kube-api-access-xtcl2\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.068405 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.068349    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30709861-ba73-4a0e-8306-5f3e1b991dee-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.068466 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.068448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30709861-ba73-4a0e-8306-5f3e1b991dee-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.069107 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.069090    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:55:43.069182 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.069122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 16 21:55:43.069182 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.069122    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 16 21:55:43.168958 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.168929    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30709861-ba73-4a0e-8306-5f3e1b991dee-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.169160 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.168991    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30709861-ba73-4a0e-8306-5f3e1b991dee-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.169160 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.169020    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xtcl2\" (UniqueName: \"kubernetes.io/projected/30709861-ba73-4a0e-8306-5f3e1b991dee-kube-api-access-xtcl2\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.171489 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.171458    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/30709861-ba73-4a0e-8306-5f3e1b991dee-tmp\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.171634 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.171561    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30709861-ba73-4a0e-8306-5f3e1b991dee-tls-certs\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.264325 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.264293    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xtcl2\" (UniqueName: \"kubernetes.io/projected/30709861-ba73-4a0e-8306-5f3e1b991dee-kube-api-access-xtcl2\") pod \"kube-auth-proxy-5c8bcbdbf8-xhws9\" (UID: \"30709861-ba73-4a0e-8306-5f3e1b991dee\") " pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.364345 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.364315    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"
Jul 16 21:55:43.462653 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.462619    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"]
Jul 16 21:55:43.835392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.835333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9"]
Jul 16 21:55:43.842547 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:55:43.842519    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30709861_ba73_4a0e_8306_5f3e1b991dee.slice/crio-a0d021f36e397cd0ba55328391e56feddc60ff4f763795446e532ba4351bb891 WatchSource:0}: Error finding container a0d021f36e397cd0ba55328391e56feddc60ff4f763795446e532ba4351bb891: Status 404 returned error can't find the container with id a0d021f36e397cd0ba55328391e56feddc60ff4f763795446e532ba4351bb891
Jul 16 21:55:43.922641 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:43.922561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9" event={"ID":"30709861-ba73-4a0e-8306-5f3e1b991dee","Type":"ContainerStarted","Data":"a0d021f36e397cd0ba55328391e56feddc60ff4f763795446e532ba4351bb891"}
Jul 16 21:55:46.161223 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.161186    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fgrdt"]
Jul 16 21:55:46.163961 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.163935    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.199013 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.198986    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.199090 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.199050    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-prhxz\" (UniqueName: \"kubernetes.io/projected/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-kube-api-access-prhxz\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.222044 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.222017    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 16 21:55:46.276022 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.275987    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fgrdt"]
Jul 16 21:55:46.300187 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.300154    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-prhxz\" (UniqueName: \"kubernetes.io/projected/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-kube-api-access-prhxz\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.300392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.300265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.300392 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:55:46.300374    2581 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 16 21:55:46.300499 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:55:46.300444    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs podName:fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2 nodeName:}" failed. No retries permitted until 2026-07-16 21:55:46.800421486 +0000 UTC m=+534.129861384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs") pod "model-serving-api-54d89bd5fc-fgrdt" (UID: "fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2") : secret "model-serving-api-tls" not found
Jul 16 21:55:46.425701 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.425634    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-prhxz\" (UniqueName: \"kubernetes.io/projected/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-kube-api-access-prhxz\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.805717 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.805626    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.808317 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.808284    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2-tls-certs\") pod \"model-serving-api-54d89bd5fc-fgrdt\" (UID: \"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2\") " pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:46.935798 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.935758    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9" event={"ID":"30709861-ba73-4a0e-8306-5f3e1b991dee","Type":"ContainerStarted","Data":"aa7f10524f56bf7645eed838d4ba2b9a5816ebb2633e8d02003ff88f06af0716"}
Jul 16 21:55:46.985017 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:46.984961    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5c8bcbdbf8-xhws9" podStartSLOduration=2.630758899 podStartE2EDuration="4.984941743s" totalImagesPullingTime="2.354182844s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:46.980556966 +0000 UTC m=+534.309996869" watchObservedRunningTime="2026-07-16 21:55:46.984941743 +0000 UTC m=+534.314381647"
Jul 16 21:55:47.074154 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:47.074067    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:47.227935 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:47.227902    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-fgrdt"]
Jul 16 21:55:47.230629 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:55:47.230600    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcfd39c5_1e47_4ecc_a2bb_7c6bcaf921c2.slice/crio-35f57b31b8ae3e4094320ce90636ad197127975681cd54882941c66b98c16d3a WatchSource:0}: Error finding container 35f57b31b8ae3e4094320ce90636ad197127975681cd54882941c66b98c16d3a: Status 404 returned error can't find the container with id 35f57b31b8ae3e4094320ce90636ad197127975681cd54882941c66b98c16d3a
Jul 16 21:55:47.941298 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:47.941253    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt" event={"ID":"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2","Type":"ContainerStarted","Data":"35f57b31b8ae3e4094320ce90636ad197127975681cd54882941c66b98c16d3a"}
Jul 16 21:55:48.946671 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:48.946590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt" event={"ID":"fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2","Type":"ContainerStarted","Data":"2b7b9c9dd7cdc163514384b077bf5c04ae0ad5a8bcfdf365b666e37f824fe16b"}
Jul 16 21:55:48.947034 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:48.946694    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:55:49.076834 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:49.076783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt" podStartSLOduration=1.637609931 podStartE2EDuration="3.076767641s" totalImagesPullingTime="1.43915771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:49.075256598 +0000 UTC m=+536.404696500" watchObservedRunningTime="2026-07-16 21:55:49.076767641 +0000 UTC m=+536.406207545"
Jul 16 21:55:55.541668 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.541633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"]
Jul 16 21:55:55.547853 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.547832    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.555081 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.554743    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 21:55:55.555081 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.554816    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 21:55:55.555545 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.555519    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 21:55:55.555810 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.555785    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-vz5h5\""
Jul 16 21:55:55.555892 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.555821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 21:55:55.669678 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.669644    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"]
Jul 16 21:55:55.683667 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.683631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287d78c-ded5-4b18-896c-c64ab048bcd3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.683934 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.683911    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.684057 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.684042    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.684147 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.684135    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vn9b6\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-kube-api-access-vn9b6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.684261 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.684246    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.684392 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.684377    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.684494 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.684482    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785205 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785169    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287d78c-ded5-4b18-896c-c64ab048bcd3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785205 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785210    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785248    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785274    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vn9b6\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-kube-api-access-vn9b6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785299    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785482 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785433    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.785681 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.785484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.786179 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.786152    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.787882 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.787854    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.788121 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.788102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d287d78c-ded5-4b18-896c-c64ab048bcd3-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.788334 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.788314    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.788402 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.788332    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.796059 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.795998    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vn9b6\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-kube-api-access-vn9b6\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.803512 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.803492    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d287d78c-ded5-4b18-896c-c64ab048bcd3-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-dxnq2\" (UID: \"d287d78c-ded5-4b18-896c-c64ab048bcd3\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:55.860395 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:55.860368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:56.037099 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:55:56.037059    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd287d78c_ded5_4b18_896c_c64ab048bcd3.slice/crio-0113c75ddffbc2f70a375777163b5a2b60cfa6124be366fcdb2618aa15f1d77a WatchSource:0}: Error finding container 0113c75ddffbc2f70a375777163b5a2b60cfa6124be366fcdb2618aa15f1d77a: Status 404 returned error can't find the container with id 0113c75ddffbc2f70a375777163b5a2b60cfa6124be366fcdb2618aa15f1d77a
Jul 16 21:55:56.040612 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.040588    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"]
Jul 16 21:55:56.879050 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.879021    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"]
Jul 16 21:55:56.886221 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.886197    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.892074 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.892051    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-rp767\""
Jul 16 21:55:56.936307 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.936155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"]
Jul 16 21:55:56.978307 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.978270    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2" event={"ID":"d287d78c-ded5-4b18-896c-c64ab048bcd3","Type":"ContainerStarted","Data":"0113c75ddffbc2f70a375777163b5a2b60cfa6124be366fcdb2618aa15f1d77a"}
Jul 16 21:55:56.995941 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.995907    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6227k\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-kube-api-access-6227k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996136 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.995962    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996136 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996000    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083b09d1-de49-4950-a8fa-75b4cc05a71d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996136 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996072    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996136 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996119    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996398 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996398 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996222    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996398 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996278    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:56.996398 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:56.996307    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097436 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097448    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097475    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097503    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6227k\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-kube-api-access-6227k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097591    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097624    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083b09d1-de49-4950-a8fa-75b4cc05a71d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097924 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097655    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097924 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097703    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097924 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097883    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.097924 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.097906    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.098256 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.098202    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.098456 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.098285    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-data\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.098600 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.098577    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/083b09d1-de49-4950-a8fa-75b4cc05a71d-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.100619 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.100591    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.100773 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.100746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.143451 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.143342    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-istio-token\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.143908 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.143873    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6227k\" (UniqueName: \"kubernetes.io/projected/083b09d1-de49-4950-a8fa-75b4cc05a71d-kube-api-access-6227k\") pod \"data-science-gateway-data-science-gateway-class-6576859cf4m9wss\" (UID: \"083b09d1-de49-4950-a8fa-75b4cc05a71d\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.200562 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.200520    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:55:57.408783 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:57.408754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"]
Jul 16 21:55:58.264802 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:55:58.264768    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod083b09d1_de49_4950_a8fa_75b4cc05a71d.slice/crio-7065799f3c16e142e2f923eea2c1acc00c35e856cce0c3183885dec9e208355f WatchSource:0}: Error finding container 7065799f3c16e142e2f923eea2c1acc00c35e856cce0c3183885dec9e208355f: Status 404 returned error can't find the container with id 7065799f3c16e142e2f923eea2c1acc00c35e856cce0c3183885dec9e208355f
Jul 16 21:55:58.327990 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:58.327944    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:55:58.328069 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:58.328027    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:55:58.990324 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:58.990276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2" event={"ID":"d287d78c-ded5-4b18-896c-c64ab048bcd3","Type":"ContainerStarted","Data":"6d9f397ace80986115302be01ddb45b2df4968a607d281fce9d24f868f468252"}
Jul 16 21:55:58.990529 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:58.990402    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:55:58.991682 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:58.991642    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss" event={"ID":"083b09d1-de49-4950-a8fa-75b4cc05a71d","Type":"ContainerStarted","Data":"7065799f3c16e142e2f923eea2c1acc00c35e856cce0c3183885dec9e208355f"}
Jul 16 21:55:59.062497 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:59.062437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2" podStartSLOduration=1.774177986 podStartE2EDuration="4.062390687s" totalImagesPullingTime="2.288212701s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:59.05820713 +0000 UTC m=+546.387647033" watchObservedRunningTime="2026-07-16 21:55:59.062390687 +0000 UTC m=+546.391830591"
Jul 16 21:55:59.956190 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:55:59.956150    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-fgrdt"
Jul 16 21:56:00.008158 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.007926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-dxnq2"
Jul 16 21:56:00.127698 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.127659    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"]
Jul 16 21:56:00.132611 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.132590    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.144412 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.144385    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-p46nz\""
Jul 16 21:56:00.144694 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.144678    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 16 21:56:00.173162 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.173133    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"]
Jul 16 21:56:00.227714 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.227633    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"]
Jul 16 21:56:00.230571 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.230551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.233323 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.233298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.233453 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.233337    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z6t\" (UniqueName: \"kubernetes.io/projected/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-kube-api-access-b9z6t\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.238808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.238784    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 16 21:56:00.251186 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.251129    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-9ljr6\""
Jul 16 21:56:00.269679 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.269609    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"]
Jul 16 21:56:00.334283 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.334221    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pg276\" (UniqueName: \"kubernetes.io/projected/bad5efa6-6e90-43d3-838a-d553b568c204-kube-api-access-pg276\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.334470 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.334344    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.334470 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.334378    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.334470 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.334405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z6t\" (UniqueName: \"kubernetes.io/projected/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-kube-api-access-b9z6t\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.337646 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.337620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-cert\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.363463 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.363405    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z6t\" (UniqueName: \"kubernetes.io/projected/2f807aa4-2b3c-42c9-bb11-0c42de8ca13c-kube-api-access-b9z6t\") pod \"kserve-controller-manager-d6f4b65b8-cgdjh\" (UID: \"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.435429 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.435368    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.435610 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.435479    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pg276\" (UniqueName: \"kubernetes.io/projected/bad5efa6-6e90-43d3-838a-d553b568c204-kube-api-access-pg276\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.435610 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:56:00.435503    2581 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 16 21:56:00.435610 ip-10-0-138-223 kubenswrapper[2581]: E0716 21:56:00.435577    2581 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert podName:bad5efa6-6e90-43d3-838a-d553b568c204 nodeName:}" failed. No retries permitted until 2026-07-16 21:56:00.935555073 +0000 UTC m=+548.264994956 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert") pod "llmisvc-controller-manager-777c6bd7d5-z2hnh" (UID: "bad5efa6-6e90-43d3-838a-d553b568c204") : secret "llmisvc-webhook-server-cert" not found
Jul 16 21:56:00.444893 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.444864    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:00.492332 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.492223    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pg276\" (UniqueName: \"kubernetes.io/projected/bad5efa6-6e90-43d3-838a-d553b568c204-kube-api-access-pg276\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.940607 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.940440    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:00.944507 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:00.944478    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/bad5efa6-6e90-43d3-838a-d553b568c204-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-z2hnh\" (UID: \"bad5efa6-6e90-43d3-838a-d553b568c204\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:01.076465 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.076431    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:56:01.076858 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.076499    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:56:01.076858 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.076531    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:56:01.142390 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.142359    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:01.254811 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.254715    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"]
Jul 16 21:56:01.350147 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:01.350102    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"]
Jul 16 21:56:01.353130 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:56:01.353079    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podbad5efa6_6e90_43d3_838a_d553b568c204.slice/crio-ce044e9d93d51e066daa8581f6d47ae53f6355b34b8ab85f2cb402eb87a30fa7 WatchSource:0}: Error finding container ce044e9d93d51e066daa8581f6d47ae53f6355b34b8ab85f2cb402eb87a30fa7: Status 404 returned error can't find the container with id ce044e9d93d51e066daa8581f6d47ae53f6355b34b8ab85f2cb402eb87a30fa7
Jul 16 21:56:02.014176 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.014140    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh" event={"ID":"bad5efa6-6e90-43d3-838a-d553b568c204","Type":"ContainerStarted","Data":"ce044e9d93d51e066daa8581f6d47ae53f6355b34b8ab85f2cb402eb87a30fa7"}
Jul 16 21:56:02.015608 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.015548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh" event={"ID":"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c","Type":"ContainerStarted","Data":"cfc9fa78a89a52cafb7c5b381add4725b8bd5546989f5184171e78c5716ae349"}
Jul 16 21:56:02.017104 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.017082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss" event={"ID":"083b09d1-de49-4950-a8fa-75b4cc05a71d","Type":"ContainerStarted","Data":"f8177d319f19fea33fdeab20d72e514df16a58ced3a4b183b576d7d8715f03bb"}
Jul 16 21:56:02.122551 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.122484    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss" podStartSLOduration=3.313158117 podStartE2EDuration="6.122466073s" totalImagesPullingTime="2.809307956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:02.108135583 +0000 UTC m=+549.437575487" watchObservedRunningTime="2026-07-16 21:56:02.122466073 +0000 UTC m=+549.451905976"
Jul 16 21:56:02.201295 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.201258    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:56:02.207174 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:02.207143    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:56:03.023009 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:03.022977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:56:03.023918 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:03.023896    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-6576859cf4m9wss"
Jul 16 21:56:05.033190 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:05.033146    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh" event={"ID":"bad5efa6-6e90-43d3-838a-d553b568c204","Type":"ContainerStarted","Data":"369b878cd17389ef74e78698a93c10550402ddce1b360f50585fc94bf9917951"}
Jul 16 21:56:05.033679 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:05.033346    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:05.034741 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:05.034713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh" event={"ID":"2f807aa4-2b3c-42c9-bb11-0c42de8ca13c","Type":"ContainerStarted","Data":"da562f17e2b4b18f3b98975e7be26c98af9666723f07bab21c6653110167c3d8"}
Jul 16 21:56:05.034875 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:05.034805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:05.081927 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:05.081875    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh" podStartSLOduration=2.320096372 podStartE2EDuration="5.081861737s" totalImagesPullingTime="2.761765365s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:05.078568596 +0000 UTC m=+552.408008499" watchObservedRunningTime="2026-07-16 21:56:05.081861737 +0000 UTC m=+552.411301639"
Jul 16 21:56:36.042532 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:36.042501    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-z2hnh"
Jul 16 21:56:36.045633 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:36.045612    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh"
Jul 16 21:56:36.063349 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:36.063308    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-cgdjh" podStartSLOduration=33.266400202 podStartE2EDuration="36.0632938s" totalImagesPullingTime="2.796893598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:05.164398922 +0000 UTC m=+552.493838824" watchObservedRunningTime="2026-07-16 21:56:36.0632938 +0000 UTC m=+583.392733703"
Jul 16 21:56:54.318987 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:54.318870    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:56:54.320615 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:56:54.320468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 21:57:33.249372 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.249289    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"]
Jul 16 21:57:33.252450 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.252431    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:33.256578 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.256555    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-wtr6r\""
Jul 16 21:57:33.256578 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.256571    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 16 21:57:33.257302 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.257282    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:57:33.257416 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.257338    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:57:33.272827 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.272803    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"]
Jul 16 21:57:33.318591 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.318559    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qsn9w\" (UniqueName: \"kubernetes.io/projected/2cad4968-06e6-41c0-90f0-9443f79d4f44-kube-api-access-qsn9w\") pod \"dns-operator-controller-manager-864dd78c4c-4dfh2\" (UID: \"2cad4968-06e6-41c0-90f0-9443f79d4f44\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:33.419609 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.419582    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qsn9w\" (UniqueName: \"kubernetes.io/projected/2cad4968-06e6-41c0-90f0-9443f79d4f44-kube-api-access-qsn9w\") pod \"dns-operator-controller-manager-864dd78c4c-4dfh2\" (UID: \"2cad4968-06e6-41c0-90f0-9443f79d4f44\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:33.433659 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.433629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qsn9w\" (UniqueName: \"kubernetes.io/projected/2cad4968-06e6-41c0-90f0-9443f79d4f44-kube-api-access-qsn9w\") pod \"dns-operator-controller-manager-864dd78c4c-4dfh2\" (UID: \"2cad4968-06e6-41c0-90f0-9443f79d4f44\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:33.563099 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.563022    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:33.692632 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.692610    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"]
Jul 16 21:57:33.695326 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:57:33.695299    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cad4968_06e6_41c0_90f0_9443f79d4f44.slice/crio-09bda057b0b98897237e40b8b8261760891fc1f968243ffe8d2aac5c441e0fde WatchSource:0}: Error finding container 09bda057b0b98897237e40b8b8261760891fc1f968243ffe8d2aac5c441e0fde: Status 404 returned error can't find the container with id 09bda057b0b98897237e40b8b8261760891fc1f968243ffe8d2aac5c441e0fde
Jul 16 21:57:33.697373 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:33.697355    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:57:34.393602 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:34.393562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2" event={"ID":"2cad4968-06e6-41c0-90f0-9443f79d4f44","Type":"ContainerStarted","Data":"09bda057b0b98897237e40b8b8261760891fc1f968243ffe8d2aac5c441e0fde"}
Jul 16 21:57:36.406464 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:36.406424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2" event={"ID":"2cad4968-06e6-41c0-90f0-9443f79d4f44","Type":"ContainerStarted","Data":"01f7e0e33642ae4faca2bb4b52b664baaf7bfd2a3dd5fc9ca626e324ed516db1"}
Jul 16 21:57:36.406899 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:36.406579    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:36.424816 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:36.424768    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2" podStartSLOduration=1.7866270100000001 podStartE2EDuration="3.424753784s" totalImagesPullingTime="1.638126774s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:36.423728189 +0000 UTC m=+643.753168093" watchObservedRunningTime="2026-07-16 21:57:36.424753784 +0000 UTC m=+643.754193686"
Jul 16 21:57:43.067951 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.067916    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"]
Jul 16 21:57:43.071223 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.071206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:43.084825 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.084802    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-qw429\""
Jul 16 21:57:43.118099 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.118074    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"]
Jul 16 21:57:43.208365 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.208323    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4hx\" (UniqueName: \"kubernetes.io/projected/8a872677-006e-4af2-99cc-23f1ebaed274-kube-api-access-gw4hx\") pod \"limitador-operator-controller-manager-859d7b78bd-bhm6k\" (UID: \"8a872677-006e-4af2-99cc-23f1ebaed274\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:43.308937 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.308905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4hx\" (UniqueName: \"kubernetes.io/projected/8a872677-006e-4af2-99cc-23f1ebaed274-kube-api-access-gw4hx\") pod \"limitador-operator-controller-manager-859d7b78bd-bhm6k\" (UID: \"8a872677-006e-4af2-99cc-23f1ebaed274\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:43.321778 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.321715    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4hx\" (UniqueName: \"kubernetes.io/projected/8a872677-006e-4af2-99cc-23f1ebaed274-kube-api-access-gw4hx\") pod \"limitador-operator-controller-manager-859d7b78bd-bhm6k\" (UID: \"8a872677-006e-4af2-99cc-23f1ebaed274\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:43.380705 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.380662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:43.533867 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:43.533842    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"]
Jul 16 21:57:43.536219 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:57:43.536195    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a872677_006e_4af2_99cc_23f1ebaed274.slice/crio-d106876dd84855282b2417bc753193f8efb0dff91c05422cc064180aa53ad957 WatchSource:0}: Error finding container d106876dd84855282b2417bc753193f8efb0dff91c05422cc064180aa53ad957: Status 404 returned error can't find the container with id d106876dd84855282b2417bc753193f8efb0dff91c05422cc064180aa53ad957
Jul 16 21:57:44.442885 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:44.442836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k" event={"ID":"8a872677-006e-4af2-99cc-23f1ebaed274","Type":"ContainerStarted","Data":"d106876dd84855282b2417bc753193f8efb0dff91c05422cc064180aa53ad957"}
Jul 16 21:57:45.449717 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:45.449628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k" event={"ID":"8a872677-006e-4af2-99cc-23f1ebaed274","Type":"ContainerStarted","Data":"4d5e74279a3a5ef37c31bb60f8c7ffe223917ed7122862a438698ff2747361f8"}
Jul 16 21:57:45.450121 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:45.449763    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:57:45.470559 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:45.470502    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k" podStartSLOduration=0.905150296 podStartE2EDuration="2.470486611s" totalImagesPullingTime="1.565336315s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:45.469472761 +0000 UTC m=+652.798912664" watchObservedRunningTime="2026-07-16 21:57:45.470486611 +0000 UTC m=+652.799926594"
Jul 16 21:57:47.412934 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:47.412893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-4dfh2"
Jul 16 21:57:56.457367 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:57:56.457337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-bhm6k"
Jul 16 21:58:31.848823 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.848782    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 21:58:31.851218 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.851191    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:31.853650 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.853632    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pm9j9\""
Jul 16 21:58:31.854294 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.854280    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 21:58:31.863674 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.863651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 21:58:31.901441 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.901397    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 21:58:31.919008 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.918978    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c25947ed-c099-4728-b170-10980f34bf79-config-file\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:31.919139 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:31.919043    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbxmw\" (UniqueName: \"kubernetes.io/projected/c25947ed-c099-4728-b170-10980f34bf79-kube-api-access-gbxmw\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.020014 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.019981    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbxmw\" (UniqueName: \"kubernetes.io/projected/c25947ed-c099-4728-b170-10980f34bf79-kube-api-access-gbxmw\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.020270 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.020042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c25947ed-c099-4728-b170-10980f34bf79-config-file\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.020693 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.020672    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/c25947ed-c099-4728-b170-10980f34bf79-config-file\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.028244 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.028194    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbxmw\" (UniqueName: \"kubernetes.io/projected/c25947ed-c099-4728-b170-10980f34bf79-kube-api-access-gbxmw\") pod \"limitador-limitador-54d6d988d5-87hdc\" (UID: \"c25947ed-c099-4728-b170-10980f34bf79\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.162010 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.161918    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:32.308067 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.308037    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 21:58:32.310210 ip-10-0-138-223 kubenswrapper[2581]: W0716 21:58:32.310182    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc25947ed_c099_4728_b170_10980f34bf79.slice/crio-725458ad4c121b842b3f541adf3203f01315eb08b577abc4fdec0ba5bdb06b0b WatchSource:0}: Error finding container 725458ad4c121b842b3f541adf3203f01315eb08b577abc4fdec0ba5bdb06b0b: Status 404 returned error can't find the container with id 725458ad4c121b842b3f541adf3203f01315eb08b577abc4fdec0ba5bdb06b0b
Jul 16 21:58:32.652298 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:32.652265    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc" event={"ID":"c25947ed-c099-4728-b170-10980f34bf79","Type":"ContainerStarted","Data":"725458ad4c121b842b3f541adf3203f01315eb08b577abc4fdec0ba5bdb06b0b"}
Jul 16 21:58:35.670488 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:35.670454    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc" event={"ID":"c25947ed-c099-4728-b170-10980f34bf79","Type":"ContainerStarted","Data":"718692ee5258ae6cee90a844fc0f1d868b37d6cd18a5452b056635c494c1eda0"}
Jul 16 21:58:35.670932 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:35.670581    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:58:35.686808 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:35.686760    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc" podStartSLOduration=1.475045051 podStartE2EDuration="4.686745765s" totalImagesPullingTime="3.211700714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:35.684709235 +0000 UTC m=+703.014149150" watchObservedRunningTime="2026-07-16 21:58:35.686745765 +0000 UTC m=+703.016185668"
Jul 16 21:58:38.517626 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:38.517590    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:58:46.676805 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:58:46.676733    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-87hdc"
Jul 16 21:59:03.538375 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.538325    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-75f98f54f6-qzwfn" podUID="66c204a4-49fd-4589-ae29-b6a480b66dec" containerName="console" containerID="cri-o://0ad46b0c9bdef8d592c5c0e58b68e78abc65783e0e1592eb20c2b3b3b90fc8c8" gracePeriod=15
Jul 16 21:59:03.791907 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.791841    2581 generic.go:356] "Generic (PLEG): container finished" podID="66c204a4-49fd-4589-ae29-b6a480b66dec" containerID="0ad46b0c9bdef8d592c5c0e58b68e78abc65783e0e1592eb20c2b3b3b90fc8c8" exitCode=0
Jul 16 21:59:03.792030 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.791925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f98f54f6-qzwfn" event={"ID":"66c204a4-49fd-4589-ae29-b6a480b66dec","Type":"ContainerDied","Data":"0ad46b0c9bdef8d592c5c0e58b68e78abc65783e0e1592eb20c2b3b3b90fc8c8"}
Jul 16 21:59:03.792030 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.791986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75f98f54f6-qzwfn" event={"ID":"66c204a4-49fd-4589-ae29-b6a480b66dec","Type":"ContainerDied","Data":"26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b"}
Jul 16 21:59:03.792030 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.792006    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f10bc3ce6987ce8ee4072ee8c1a44b7a2cc9a5f8324a3e51e3d5195350531b"
Jul 16 21:59:03.797793 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.797773    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:59:03.887937 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.887907    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888092 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.887950    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888092 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.887987    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888092 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888092 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888053    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888309 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888121    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5kw68\" (UniqueName: \"kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888309 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert\") pod \"66c204a4-49fd-4589-ae29-b6a480b66dec\" (UID: \"66c204a4-49fd-4589-ae29-b6a480b66dec\") "
Jul 16 21:59:03.888589 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888558    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:59:03.888589 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888575    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca" (OuterVolumeSpecName: "service-ca") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:59:03.888745 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888571    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:59:03.888781 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.888694    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config" (OuterVolumeSpecName: "console-config") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:59:03.890336 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.890307    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:59:03.890572 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.890552    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:59:03.890659 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.890643    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68" (OuterVolumeSpecName: "kube-api-access-5kw68") pod "66c204a4-49fd-4589-ae29-b6a480b66dec" (UID: "66c204a4-49fd-4589-ae29-b6a480b66dec"). InnerVolumeSpecName "kube-api-access-5kw68". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:59:03.989618 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989583    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kw68\" (UniqueName: \"kubernetes.io/projected/66c204a4-49fd-4589-ae29-b6a480b66dec-kube-api-access-5kw68\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989618 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989612    2581 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989618 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989621    2581 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-trusted-ca-bundle\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989618 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989629    2581 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/66c204a4-49fd-4589-ae29-b6a480b66dec-console-oauth-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989871 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989638    2581 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989871 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989646    2581 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-oauth-serving-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:03.989871 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:03.989654    2581 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/66c204a4-49fd-4589-ae29-b6a480b66dec-console-config\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 21:59:04.795814 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:04.795785    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75f98f54f6-qzwfn"
Jul 16 21:59:04.816288 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:04.816260    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:59:04.823698 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:04.823672    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-75f98f54f6-qzwfn"]
Jul 16 21:59:05.159728 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:05.159695    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66c204a4-49fd-4589-ae29-b6a480b66dec" path="/var/lib/kubelet/pods/66c204a4-49fd-4589-ae29-b6a480b66dec/volumes"
Jul 16 21:59:51.106498 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:51.106465    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 21:59:53.066082 ip-10-0-138-223 kubenswrapper[2581]: I0716 21:59:53.066053    2581 scope.go:117] "RemoveContainer" containerID="0ad46b0c9bdef8d592c5c0e58b68e78abc65783e0e1592eb20c2b3b3b90fc8c8"
Jul 16 22:00:13.644871 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:00:13.644838    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:00:25.504639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:00:25.504562    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:00:39.701714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:00:39.701670    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:00:54.402591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:00:54.402560    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:01:02.314680 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:01:02.314639    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:01:12.099687 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:01:12.099652    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:01:54.342680 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:01:54.342652    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:01:54.345260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:01:54.344886    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:03:00.420992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:03:00.420956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:03:10.479642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:03:10.479611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:03:21.718284 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:03:21.718204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:05:01.029193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:05:01.029116    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:05:11.105653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:05:11.105615    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:05:48.199633 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:05:48.199591    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:05:57.513990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:05:57.513953    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:09.610239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:09.610194    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:18.306900 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:18.306815    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:27.607319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:27.607285    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:37.913314 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:37.908858    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:50.410880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:50.410843    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:06:54.366535 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:54.366438    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:06:54.370560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:06:54.369596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:07:00.210304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:07:00.210264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:07:21.615604 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:07:21.615559    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:07:30.813706 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:07:30.813666    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:07:53.897205 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:07:53.897125    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:08:03.803613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:08:03.803577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:08:22.717641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:08:22.717608    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:08:32.012015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:08:32.011980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:08:41.405299 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:08:41.405263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:08:51.102906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:08:51.102861    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:09:00.606587 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:09:00.606549    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:10:54.002906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:10:54.002826    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:05.506047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:05.506002    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:14.115957 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:14.115918    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:28.727710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:28.727673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:38.412871 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:38.412830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:48.116257 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:48.116199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:11:54.390975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:54.390859    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:11:54.395285 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:54.395270    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:11:56.807216 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:11:56.807177    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:12:07.104660 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:12:07.104622    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:12:17.510847 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:12:17.510762    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:12:28.204349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:12:28.204310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:12:39.710039 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:12:39.709995    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:02.411760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:02.411722    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:12.311566 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:12.311528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:22.405995 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:22.405956    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:31.906043 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:31.906010    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:38.814489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:38.814449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:13:54.224169 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:13:54.224092    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:04.710790 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:04.710753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:14.539414 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:14.539375    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:24.530402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:24.530362    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:34.215487 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:34.215445    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:43.744348 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:43.744314    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:14:57.618703 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:14:57.618673    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:15:00.164964 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.164931    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"]
Jul 16 22:15:00.165359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.165245    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66c204a4-49fd-4589-ae29-b6a480b66dec" containerName="console"
Jul 16 22:15:00.165359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.165258    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66c204a4-49fd-4589-ae29-b6a480b66dec" containerName="console"
Jul 16 22:15:00.165359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.165326    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66c204a4-49fd-4589-ae29-b6a480b66dec" containerName="console"
Jul 16 22:15:00.168349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.168332    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"
Jul 16 22:15:00.171725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.171706    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:15:00.177698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.177683    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"
Jul 16 22:15:00.190287 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.190205    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"]
Jul 16 22:15:00.323793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.323763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"]
Jul 16 22:15:00.326773 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:15:00.326736    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfea86aeb_6ba0_415b_bcfa_20e0cc58c276.slice/crio-88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971 WatchSource:0}: Error finding container 88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971: Status 404 returned error can't find the container with id 88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971
Jul 16 22:15:00.328591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.328575    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:15:00.827975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:00.827942    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w" event={"ID":"fea86aeb-6ba0-415b-bcfa-20e0cc58c276","Type":"ContainerStarted","Data":"88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971"}
Jul 16 22:15:02.839356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:02.839313    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w" event={"ID":"fea86aeb-6ba0-415b-bcfa-20e0cc58c276","Type":"ContainerStarted","Data":"14e33fd0b25aebc30edcf75edd48d7ad45058db27ecbc43e7f38c49502a89a3f"}
Jul 16 22:15:02.859290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:02.859221    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w" podStartSLOduration=0.816504883 podStartE2EDuration="2.859204234s" totalImagesPullingTime="2.042699351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:15:02.857782838 +0000 UTC m=+1690.187222741" watchObservedRunningTime="2026-07-16 22:15:02.859204234 +0000 UTC m=+1690.188644137"
Jul 16 22:15:03.845299 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:03.845265    2581 generic.go:356] "Generic (PLEG): container finished" podID="fea86aeb-6ba0-415b-bcfa-20e0cc58c276" containerID="14e33fd0b25aebc30edcf75edd48d7ad45058db27ecbc43e7f38c49502a89a3f" exitCode=0
Jul 16 22:15:03.845687 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:03.845362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w" event={"ID":"fea86aeb-6ba0-415b-bcfa-20e0cc58c276","Type":"ContainerDied","Data":"14e33fd0b25aebc30edcf75edd48d7ad45058db27ecbc43e7f38c49502a89a3f"}
Jul 16 22:15:04.978110 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:04.978087    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"
Jul 16 22:15:05.854865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:05.854837    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w"
Jul 16 22:15:05.855030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:05.854835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737335-7db6w" event={"ID":"fea86aeb-6ba0-415b-bcfa-20e0cc58c276","Type":"ContainerDied","Data":"88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971"}
Jul 16 22:15:05.855030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:05.854971    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="88d86f5032a54be4fa3a50721bdf7fa607f38a570f12504f29cb5cd814733971"
Jul 16 22:15:07.445010 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:07.444978    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:15:16.707981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:16.707901    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:15:29.711181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:29.711148    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:15:39.311497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:39.311460    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:15:51.415389 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:15:51.415350    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:05.909867 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:05.909830    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:17.627972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:17.627936    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:27.411822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:27.411786    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:40.117074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:40.117039    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:52.634131 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:52.634091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:16:54.415053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:54.414938    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:16:54.420859 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:54.419951    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:16:59.866445 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.866409    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:16:59.866824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.866727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fea86aeb-6ba0-415b-bcfa-20e0cc58c276" containerName="cleanup"
Jul 16 22:16:59.866824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.866737    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fea86aeb-6ba0-415b-bcfa-20e0cc58c276" containerName="cleanup"
Jul 16 22:16:59.866824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.866784    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fea86aeb-6ba0-415b-bcfa-20e0cc58c276" containerName="cleanup"
Jul 16 22:16:59.869588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.869562    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:16:59.875243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.875202    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-af8fc2dc-openshift-default-dockercfg-2tb9l\""
Jul 16 22:16:59.939486 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:16:59.939457    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:17:00.020651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020613    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020658    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020693    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020746    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020761    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020776    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.020813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.021029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020820    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.021029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.020873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8d4q\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.121906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.121873    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.121906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.121905    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.121921    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.121939    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.121965    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122012    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8d4q\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122053    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122102    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122430    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122532    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.122939 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.122917    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.124596 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.124578    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.124691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.124673    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.137543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.137519    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.140868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.140845    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8d4q\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q\") pod \"e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") " pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.178894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.178862    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:00.323328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.323286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:17:00.328277 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:17:00.328224    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a366b4d_a4f3_4a75_bb2f_23302e3be8a7.slice/crio-cb26e626701fd1ce06e653aff9f5fdd3271deed568dd03bb660342cc855ce39e WatchSource:0}: Error finding container cb26e626701fd1ce06e653aff9f5fdd3271deed568dd03bb660342cc855ce39e: Status 404 returned error can't find the container with id cb26e626701fd1ce06e653aff9f5fdd3271deed568dd03bb660342cc855ce39e
Jul 16 22:17:00.330267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.330198    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:00.330367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.330322    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:00.330429 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.330368    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:00.348486 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:00.348449    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerStarted","Data":"cb26e626701fd1ce06e653aff9f5fdd3271deed568dd03bb660342cc855ce39e"}
Jul 16 22:17:01.354913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:01.354873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerStarted","Data":"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"}
Jul 16 22:17:01.394455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:01.394378    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podStartSLOduration=2.394358435 podStartE2EDuration="2.394358435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:01.391027321 +0000 UTC m=+1808.720467239" watchObservedRunningTime="2026-07-16 22:17:01.394358435 +0000 UTC m=+1808.723798329"
Jul 16 22:17:02.179074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:02.179035    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:02.184519 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:02.184480    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:02.359470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:02.359438    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:02.360339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:02.360319    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:05.912760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:05.912729    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:17:06.032783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.032753    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:06.036007 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.035991    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:06.039368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.039339    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:17:06.039533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.039498    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:17:06.046357 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.046330    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:06.175961 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.175873    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2\" (UID: \"f4b4b5f0-56e3-4298-8d46-f725a8257bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:06.277039 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.277002    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2\" (UID: \"f4b4b5f0-56e3-4298-8d46-f725a8257bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:06.277621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.277598    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2\" (UID: \"f4b4b5f0-56e3-4298-8d46-f725a8257bff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:06.349069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.349032    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:06.480066 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:06.480043    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:06.482342 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:17:06.482318    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b4b5f0_56e3_4298_8d46_f725a8257bff.slice/crio-9735b2f349da3a33c1c8aeedf4a9d5c95012376fe557752e4aa9d8fd3e9a5ec7 WatchSource:0}: Error finding container 9735b2f349da3a33c1c8aeedf4a9d5c95012376fe557752e4aa9d8fd3e9a5ec7: Status 404 returned error can't find the container with id 9735b2f349da3a33c1c8aeedf4a9d5c95012376fe557752e4aa9d8fd3e9a5ec7
Jul 16 22:17:07.382788 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:07.382754    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerID="611c858d29d490e7b2a6ab2d8d1b581e3b1149a21560ec8517cd7947c875abb7" exitCode=7
Jul 16 22:17:07.383162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:07.382850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerDied","Data":"611c858d29d490e7b2a6ab2d8d1b581e3b1149a21560ec8517cd7947c875abb7"}
Jul 16 22:17:07.383162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:07.382900    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerStarted","Data":"9735b2f349da3a33c1c8aeedf4a9d5c95012376fe557752e4aa9d8fd3e9a5ec7"}
Jul 16 22:17:07.383162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:07.382992    2581 scope.go:117] "RemoveContainer" containerID="611c858d29d490e7b2a6ab2d8d1b581e3b1149a21560ec8517cd7947c875abb7"
Jul 16 22:17:08.388948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:08.388919    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerID="c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6" exitCode=7
Jul 16 22:17:08.389360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:08.389005    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerDied","Data":"c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6"}
Jul 16 22:17:08.389360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:08.389076    2581 scope.go:117] "RemoveContainer" containerID="611c858d29d490e7b2a6ab2d8d1b581e3b1149a21560ec8517cd7947c875abb7"
Jul 16 22:17:08.389360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:08.389177    2581 scope.go:117] "RemoveContainer" containerID="c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6"
Jul 16 22:17:08.389486 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:17:08.389448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2_odh-ai-gateway-infra(f4b4b5f0-56e3-4298-8d46-f725a8257bff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff"
Jul 16 22:17:09.395451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:09.395421    2581 scope.go:117] "RemoveContainer" containerID="c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6"
Jul 16 22:17:09.395969 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:17:09.395687    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2_odh-ai-gateway-infra(f4b4b5f0-56e3-4298-8d46-f725a8257bff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff"
Jul 16 22:17:13.419055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.418963    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerID="cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552" exitCode=0
Jul 16 22:17:13.419055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.419033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerDied","Data":"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"}
Jul 16 22:17:13.419524 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.419487    2581 scope.go:117] "RemoveContainer" containerID="cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"
Jul 16 22:17:13.420461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.420434    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:13.420596 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.420505    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:13.420596 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:13.420534    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:17:14.426039 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:14.425998    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerStarted","Data":"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"}
Jul 16 22:17:15.178997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:15.178959    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:15.180345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:15.180317    2581 patch_prober.go:28] interesting pod/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Jul 16 22:17:15.180477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:15.180370    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Jul 16 22:17:16.179731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:16.179691    2581 patch_prober.go:28] interesting pod/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Jul 16 22:17:16.180118 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:16.179746    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Jul 16 22:17:17.179514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:17.179474    2581 patch_prober.go:28] interesting pod/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Jul 16 22:17:17.179698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:17.179531    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Jul 16 22:17:17.359804 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:17.359753    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:18.180339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:18.180294    2581 patch_prober.go:28] interesting pod/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused" start-of-body=
Jul 16 22:17:18.180545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:18.180360    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.28:15021/healthz/ready\": dial tcp 10.133.0.28:15021: connect: connection refused"
Jul 16 22:17:19.183326 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:19.183299    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:19.184358 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:19.184337    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:17:24.155070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:24.155035    2581 scope.go:117] "RemoveContainer" containerID="c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6"
Jul 16 22:17:24.471846 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:24.471818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerStarted","Data":"ea2edba7957605b65066dc0590a78270ff4e21ce2e8377ba1678681998dc4197"}
Jul 16 22:17:24.489058 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:24.489008    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" podStartSLOduration=18.488993658 podStartE2EDuration="18.488993658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:24.487499858 +0000 UTC m=+1831.816939761" watchObservedRunningTime="2026-07-16 22:17:24.488993658 +0000 UTC m=+1831.818433558"
Jul 16 22:17:25.477698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:25.477663    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerID="ea2edba7957605b65066dc0590a78270ff4e21ce2e8377ba1678681998dc4197" exitCode=7
Jul 16 22:17:25.478081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:25.477735    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerDied","Data":"ea2edba7957605b65066dc0590a78270ff4e21ce2e8377ba1678681998dc4197"}
Jul 16 22:17:25.478081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:25.477790    2581 scope.go:117] "RemoveContainer" containerID="c4d295aefade6b87c74a838c4096591bc4500f97705d3c504926960ae96af8a6"
Jul 16 22:17:25.478081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:25.478080    2581 scope.go:117] "RemoveContainer" containerID="ea2edba7957605b65066dc0590a78270ff4e21ce2e8377ba1678681998dc4197"
Jul 16 22:17:25.478376 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:17:25.478351    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2_odh-ai-gateway-infra(f4b4b5f0-56e3-4298-8d46-f725a8257bff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff"
Jul 16 22:17:25.510167 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:25.510133    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:26.615501 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:26.615474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:26.659578 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:26.659546    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca\") pod \"f4b4b5f0-56e3-4298-8d46-f725a8257bff\" (UID: \"f4b4b5f0-56e3-4298-8d46-f725a8257bff\") "
Jul 16 22:17:26.659887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:26.659864    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f4b4b5f0-56e3-4298-8d46-f725a8257bff" (UID: "f4b4b5f0-56e3-4298-8d46-f725a8257bff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:26.760616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:26.760537    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f4b4b5f0-56e3-4298-8d46-f725a8257bff-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:17:27.488910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:27.488832    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"
Jul 16 22:17:27.488910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:27.488865    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2" event={"ID":"f4b4b5f0-56e3-4298-8d46-f725a8257bff","Type":"ContainerDied","Data":"9735b2f349da3a33c1c8aeedf4a9d5c95012376fe557752e4aa9d8fd3e9a5ec7"}
Jul 16 22:17:27.489087 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:27.488925    2581 scope.go:117] "RemoveContainer" containerID="ea2edba7957605b65066dc0590a78270ff4e21ce2e8377ba1678681998dc4197"
Jul 16 22:17:27.505296 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:27.505273    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:27.510019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:27.509995    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhvv2"]
Jul 16 22:17:29.160297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:17:29.160263    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" path="/var/lib/kubelet/pods/f4b4b5f0-56e3-4298-8d46-f725a8257bff/volumes"
Jul 16 22:18:06.078394 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078358    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078716    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078728    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078746    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078752    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078760    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078833    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.078839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078840    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.079072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.078846    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4b4b5f0-56e3-4298-8d46-f725a8257bff" containerName="revoke-keys"
Jul 16 22:18:06.081796 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.081776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:06.084187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.084165    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:06.084288 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.084217    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:18:06.090015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.089992    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:06.092577 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.092551    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj\" (UID: \"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:06.193496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.193453    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj\" (UID: \"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:06.194063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.194038    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj\" (UID: \"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:06.393264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.393208    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:06.523626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.523598    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:06.525553 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:18:06.525515    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda4dc5e9_e6cb_4ebc_b592_bf2ea514efea.slice/crio-32e61cd0a3e290a749fd005cbc369178900046fe0b03d08a34be5a8709f9043b WatchSource:0}: Error finding container 32e61cd0a3e290a749fd005cbc369178900046fe0b03d08a34be5a8709f9043b: Status 404 returned error can't find the container with id 32e61cd0a3e290a749fd005cbc369178900046fe0b03d08a34be5a8709f9043b
Jul 16 22:18:06.665147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:06.665067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerStarted","Data":"32e61cd0a3e290a749fd005cbc369178900046fe0b03d08a34be5a8709f9043b"}
Jul 16 22:18:07.670295 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:07.670259    2581 generic.go:356] "Generic (PLEG): container finished" podID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerID="bf0840836acd76a9d5b8a3dbc6c9b7c295610960d0a956f56c4543e17c86f464" exitCode=7
Jul 16 22:18:07.670691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:07.670312    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerDied","Data":"bf0840836acd76a9d5b8a3dbc6c9b7c295610960d0a956f56c4543e17c86f464"}
Jul 16 22:18:07.670691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:07.670573    2581 scope.go:117] "RemoveContainer" containerID="bf0840836acd76a9d5b8a3dbc6c9b7c295610960d0a956f56c4543e17c86f464"
Jul 16 22:18:08.676357 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:08.676318    2581 generic.go:356] "Generic (PLEG): container finished" podID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerID="dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d" exitCode=7
Jul 16 22:18:08.676799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:08.676369    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerDied","Data":"dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d"}
Jul 16 22:18:08.676799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:08.676416    2581 scope.go:117] "RemoveContainer" containerID="bf0840836acd76a9d5b8a3dbc6c9b7c295610960d0a956f56c4543e17c86f464"
Jul 16 22:18:08.676799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:08.676597    2581 scope.go:117] "RemoveContainer" containerID="dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d"
Jul 16 22:18:08.676923 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:18:08.676822    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj_odh-ai-gateway-infra(da4dc5e9-e6cb-4ebc-b592-bf2ea514efea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea"
Jul 16 22:18:09.682183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:09.682149    2581 scope.go:117] "RemoveContainer" containerID="dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d"
Jul 16 22:18:09.682618 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:18:09.682446    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj_odh-ai-gateway-infra(da4dc5e9-e6cb-4ebc-b592-bf2ea514efea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea"
Jul 16 22:18:23.158622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:23.158551    2581 scope.go:117] "RemoveContainer" containerID="dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d"
Jul 16 22:18:23.744498 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:23.744412    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerStarted","Data":"b9632a7abdcb8a28a7e6e6bfe0e45ae3c6896d1e255f3aaf0e3f4b14893a0202"}
Jul 16 22:18:23.764085 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:23.764042    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" podStartSLOduration=17.764026922 podStartE2EDuration="17.764026922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:18:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:18:23.762544966 +0000 UTC m=+1891.091984870" watchObservedRunningTime="2026-07-16 22:18:23.764026922 +0000 UTC m=+1891.093466803"
Jul 16 22:18:24.750748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:24.750715    2581 generic.go:356] "Generic (PLEG): container finished" podID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerID="b9632a7abdcb8a28a7e6e6bfe0e45ae3c6896d1e255f3aaf0e3f4b14893a0202" exitCode=7
Jul 16 22:18:24.751199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:24.750782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerDied","Data":"b9632a7abdcb8a28a7e6e6bfe0e45ae3c6896d1e255f3aaf0e3f4b14893a0202"}
Jul 16 22:18:24.751199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:24.750844    2581 scope.go:117] "RemoveContainer" containerID="dc30a017bd50b4f672912677e891fd6259842b613faac34552437648ea91b47d"
Jul 16 22:18:24.751199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:24.751150    2581 scope.go:117] "RemoveContainer" containerID="b9632a7abdcb8a28a7e6e6bfe0e45ae3c6896d1e255f3aaf0e3f4b14893a0202"
Jul 16 22:18:24.751451 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:18:24.751429    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj_odh-ai-gateway-infra(da4dc5e9-e6cb-4ebc-b592-bf2ea514efea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea"
Jul 16 22:18:24.773928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:24.773908    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:25.887656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:25.887632    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:25.959107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:25.959075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca\") pod \"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea\" (UID: \"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea\") "
Jul 16 22:18:25.959489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:25.959465    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" (UID: "da4dc5e9-e6cb-4ebc-b592-bf2ea514efea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:18:26.060764 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.060678    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:18:26.762367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.762336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"
Jul 16 22:18:26.762367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.762352    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj" event={"ID":"da4dc5e9-e6cb-4ebc-b592-bf2ea514efea","Type":"ContainerDied","Data":"32e61cd0a3e290a749fd005cbc369178900046fe0b03d08a34be5a8709f9043b"}
Jul 16 22:18:26.762621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.762431    2581 scope.go:117] "RemoveContainer" containerID="b9632a7abdcb8a28a7e6e6bfe0e45ae3c6896d1e255f3aaf0e3f4b14893a0202"
Jul 16 22:18:26.788198 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.788168    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:26.789549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:26.789529    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rffbj"]
Jul 16 22:18:27.161842 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:18:27.161811    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" path="/var/lib/kubelet/pods/da4dc5e9-e6cb-4ebc-b592-bf2ea514efea/volumes"
Jul 16 22:19:06.152405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152370    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152695    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152706    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152733    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152785    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152794    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.152860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.152800    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:19:06.155754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.155731    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:06.159971 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.159952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:19:06.160577 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.160563    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:19:06.171631 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.171611    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:06.205294 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.205244    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm\" (UID: \"9471607b-a6e5-4b43-acbd-d5a5c1274096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:06.306583 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.306546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm\" (UID: \"9471607b-a6e5-4b43-acbd-d5a5c1274096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:06.307106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.307089    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm\" (UID: \"9471607b-a6e5-4b43-acbd-d5a5c1274096\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:06.466085 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.466012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:06.608736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.608702    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:06.611778 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:19:06.611753    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9471607b_a6e5_4b43_acbd_d5a5c1274096.slice/crio-b60650ff58b67173cc52c8a273d2bbeaf045c2b717d6ad015e6a67f4ce1a9283 WatchSource:0}: Error finding container b60650ff58b67173cc52c8a273d2bbeaf045c2b717d6ad015e6a67f4ce1a9283: Status 404 returned error can't find the container with id b60650ff58b67173cc52c8a273d2bbeaf045c2b717d6ad015e6a67f4ce1a9283
Jul 16 22:19:06.935793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.935761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerStarted","Data":"33b286477edf541b062be3e73b59dc69f57206edbf9d27c767e162df313ccebd"}
Jul 16 22:19:06.935922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.935809    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerStarted","Data":"b60650ff58b67173cc52c8a273d2bbeaf045c2b717d6ad015e6a67f4ce1a9283"}
Jul 16 22:19:06.990569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:06.990519    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" podStartSLOduration=0.990504015 podStartE2EDuration="990.504015ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:06.9876867 +0000 UTC m=+1934.317126603" watchObservedRunningTime="2026-07-16 22:19:06.990504015 +0000 UTC m=+1934.319943919"
Jul 16 22:19:07.942028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:07.941994    2581 generic.go:356] "Generic (PLEG): container finished" podID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerID="33b286477edf541b062be3e73b59dc69f57206edbf9d27c767e162df313ccebd" exitCode=7
Jul 16 22:19:07.942453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:07.942086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerDied","Data":"33b286477edf541b062be3e73b59dc69f57206edbf9d27c767e162df313ccebd"}
Jul 16 22:19:07.942453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:07.942284    2581 scope.go:117] "RemoveContainer" containerID="33b286477edf541b062be3e73b59dc69f57206edbf9d27c767e162df313ccebd"
Jul 16 22:19:08.948494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:08.948457    2581 generic.go:356] "Generic (PLEG): container finished" podID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerID="46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01" exitCode=7
Jul 16 22:19:08.948886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:08.948551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerDied","Data":"46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01"}
Jul 16 22:19:08.948886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:08.948604    2581 scope.go:117] "RemoveContainer" containerID="33b286477edf541b062be3e73b59dc69f57206edbf9d27c767e162df313ccebd"
Jul 16 22:19:08.948886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:08.948736    2581 scope.go:117] "RemoveContainer" containerID="46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01"
Jul 16 22:19:08.949033 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:19:08.948952    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm_odh-ai-gateway-infra(9471607b-a6e5-4b43-acbd-d5a5c1274096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096"
Jul 16 22:19:09.955405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:09.955379    2581 scope.go:117] "RemoveContainer" containerID="46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01"
Jul 16 22:19:09.955837 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:19:09.955634    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm_odh-ai-gateway-infra(9471607b-a6e5-4b43-acbd-d5a5c1274096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096"
Jul 16 22:19:25.155120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:25.155090    2581 scope.go:117] "RemoveContainer" containerID="46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01"
Jul 16 22:19:26.025040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:26.025003    2581 generic.go:356] "Generic (PLEG): container finished" podID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerID="ef3fb2d9755183bc345c670c37ffc3fbe9bf767b819d6cf3b6558e35a2b44f73" exitCode=7
Jul 16 22:19:26.025241 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:26.025082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerDied","Data":"ef3fb2d9755183bc345c670c37ffc3fbe9bf767b819d6cf3b6558e35a2b44f73"}
Jul 16 22:19:26.025241 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:26.025160    2581 scope.go:117] "RemoveContainer" containerID="46de200cd15f9309b697546ac1be528b030c95965ef6c296fd18047a5eac9c01"
Jul 16 22:19:26.025522 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:26.025499    2581 scope.go:117] "RemoveContainer" containerID="ef3fb2d9755183bc345c670c37ffc3fbe9bf767b819d6cf3b6558e35a2b44f73"
Jul 16 22:19:26.025734 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:19:26.025712    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm_odh-ai-gateway-infra(9471607b-a6e5-4b43-acbd-d5a5c1274096)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096"
Jul 16 22:19:27.078942 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:27.078913    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:27.211576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:27.211554    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:27.277962 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:27.277931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca\") pod \"9471607b-a6e5-4b43-acbd-d5a5c1274096\" (UID: \"9471607b-a6e5-4b43-acbd-d5a5c1274096\") "
Jul 16 22:19:27.278329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:27.278304    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9471607b-a6e5-4b43-acbd-d5a5c1274096" (UID: "9471607b-a6e5-4b43-acbd-d5a5c1274096"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:27.378536 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:27.378510    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9471607b-a6e5-4b43-acbd-d5a5c1274096-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:19:28.036832 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:28.036803    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"
Jul 16 22:19:28.037000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:28.036802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm" event={"ID":"9471607b-a6e5-4b43-acbd-d5a5c1274096","Type":"ContainerDied","Data":"b60650ff58b67173cc52c8a273d2bbeaf045c2b717d6ad015e6a67f4ce1a9283"}
Jul 16 22:19:28.037000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:28.036950    2581 scope.go:117] "RemoveContainer" containerID="ef3fb2d9755183bc345c670c37ffc3fbe9bf767b819d6cf3b6558e35a2b44f73"
Jul 16 22:19:28.069742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:28.069707    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:28.095928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:28.095898    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-28nqm"]
Jul 16 22:19:29.160542 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:19:29.160510    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" path="/var/lib/kubelet/pods/9471607b-a6e5-4b43-acbd-d5a5c1274096/volumes"
Jul 16 22:20:06.145062 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.145030    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:20:06.145622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.145260    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy" containerID="cri-o://24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310" gracePeriod=30
Jul 16 22:20:06.190934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.190902    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:06.191205 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191194    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191207    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191238    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191244    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191258    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:20:06.191309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191264    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da4dc5e9-e6cb-4ebc-b592-bf2ea514efea" containerName="revoke-keys"
Jul 16 22:20:06.191469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191314    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191323    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.191469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.191328    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:06.194315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.194297    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:06.196780 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.196754    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:20:06.196780 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.196754    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:20:06.200946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.200927    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:06.302875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.302837    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw\" (UID: \"96a728ea-775e-4085-ac1c-2d98f1940c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:06.404166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.404094    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw\" (UID: \"96a728ea-775e-4085-ac1c-2d98f1940c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:06.404675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.404657    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw\" (UID: \"96a728ea-775e-4085-ac1c-2d98f1940c1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:06.506758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.506734    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:06.839451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:06.839428    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:06.841721 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:20:06.841696    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96a728ea_775e_4085_ac1c_2d98f1940c1a.slice/crio-59ed5f30541ea19a6b13188688b536f43ababa5a1665d4978066414a48743ad2 WatchSource:0}: Error finding container 59ed5f30541ea19a6b13188688b536f43ababa5a1665d4978066414a48743ad2: Status 404 returned error can't find the container with id 59ed5f30541ea19a6b13188688b536f43ababa5a1665d4978066414a48743ad2
Jul 16 22:20:07.203505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:07.203419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerStarted","Data":"45bbc75ad8ccc5f64ff03dafc194a92550fbed943dccde5d6fa6346cd286b61b"}
Jul 16 22:20:07.203505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:07.203483    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerStarted","Data":"59ed5f30541ea19a6b13188688b536f43ababa5a1665d4978066414a48743ad2"}
Jul 16 22:20:07.219761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:07.219716    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" podStartSLOduration=1.219699365 podStartE2EDuration="1.219699365s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:07.218272719 +0000 UTC m=+1994.547712621" watchObservedRunningTime="2026-07-16 22:20:07.219699365 +0000 UTC m=+1994.549139262"
Jul 16 22:20:08.208823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:08.208789    2581 generic.go:356] "Generic (PLEG): container finished" podID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerID="45bbc75ad8ccc5f64ff03dafc194a92550fbed943dccde5d6fa6346cd286b61b" exitCode=7
Jul 16 22:20:08.209200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:08.208839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerDied","Data":"45bbc75ad8ccc5f64ff03dafc194a92550fbed943dccde5d6fa6346cd286b61b"}
Jul 16 22:20:08.209200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:08.209099    2581 scope.go:117] "RemoveContainer" containerID="45bbc75ad8ccc5f64ff03dafc194a92550fbed943dccde5d6fa6346cd286b61b"
Jul 16 22:20:09.214573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:09.214492    2581 generic.go:356] "Generic (PLEG): container finished" podID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerID="d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92" exitCode=7
Jul 16 22:20:09.214573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:09.214562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerDied","Data":"d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92"}
Jul 16 22:20:09.214981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:09.214604    2581 scope.go:117] "RemoveContainer" containerID="45bbc75ad8ccc5f64ff03dafc194a92550fbed943dccde5d6fa6346cd286b61b"
Jul 16 22:20:09.214981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:09.214752    2581 scope.go:117] "RemoveContainer" containerID="d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92"
Jul 16 22:20:09.214981 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:09.214946    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw_odh-ai-gateway-infra(96a728ea-775e-4085-ac1c-2d98f1940c1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a"
Jul 16 22:20:09.303223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:09.303195    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:20:10.220690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:10.220663    2581 scope.go:117] "RemoveContainer" containerID="d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92"
Jul 16 22:20:10.221135 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:10.220925    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw_odh-ai-gateway-infra(96a728ea-775e-4085-ac1c-2d98f1940c1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a"
Jul 16 22:20:11.394621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.394596    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:20:11.446030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.445996    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d8d4q\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446063    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446086    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446152    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446180    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446256    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446291    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs\") pod \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\" (UID: \"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7\") "
Jul 16 22:20:11.446617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446466    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:11.446617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446543    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.446720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446625    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:11.446720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446680    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:11.446720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.446690    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:11.447276 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.447211    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data" (OuterVolumeSpecName: "istio-data") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:11.448464 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.448443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q" (OuterVolumeSpecName: "kube-api-access-d8d4q") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "kube-api-access-d8d4q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:20:11.448566 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.448531    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:20:11.448958 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.448931    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token" (OuterVolumeSpecName: "istio-token") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:20:11.449042 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.448943    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" (UID: "4a366b4d-a4f3-4a75-bb2f-23302e3be8a7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:20:11.546963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546930    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8d4q\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-kube-api-access-d8d4q\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.546963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546959    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.546963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546968    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.547179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546977    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.547179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546984    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.547179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.546992    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.547179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.547000    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:11.547179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:11.547007    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:12.230250 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.230191    2581 generic.go:356] "Generic (PLEG): container finished" podID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerID="24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310" exitCode=0
Jul 16 22:20:12.230417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.230263    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"
Jul 16 22:20:12.230417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.230271    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerDied","Data":"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"}
Jul 16 22:20:12.230417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.230327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w" event={"ID":"4a366b4d-a4f3-4a75-bb2f-23302e3be8a7","Type":"ContainerDied","Data":"cb26e626701fd1ce06e653aff9f5fdd3271deed568dd03bb660342cc855ce39e"}
Jul 16 22:20:12.230417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.230348    2581 scope.go:117] "RemoveContainer" containerID="24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"
Jul 16 22:20:12.239454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.239437    2581 scope.go:117] "RemoveContainer" containerID="cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"
Jul 16 22:20:12.246610 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.246592    2581 scope.go:117] "RemoveContainer" containerID="24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"
Jul 16 22:20:12.246859 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:12.246837    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310\": container with ID starting with 24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310 not found: ID does not exist" containerID="24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"
Jul 16 22:20:12.246950 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.246864    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310"} err="failed to get container status \"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310\": rpc error: code = NotFound desc = could not find container \"24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310\": container with ID starting with 24e8a781c436e0621b9cee21e4eb297518a26cfd87ec27cb19e51c5d558ca310 not found: ID does not exist"
Jul 16 22:20:12.246950 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.246881    2581 scope.go:117] "RemoveContainer" containerID="cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"
Jul 16 22:20:12.247102 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:12.247085    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552\": container with ID starting with cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552 not found: ID does not exist" containerID="cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"
Jul 16 22:20:12.247146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.247108    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552"} err="failed to get container status \"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552\": rpc error: code = NotFound desc = could not find container \"cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552\": container with ID starting with cf8ac399cea4e919bd5f643b5c3b834892467c2f692f400b8404bbfec3d1e552 not found: ID does not exist"
Jul 16 22:20:12.251499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.251480    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:20:12.255548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:12.255528    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-af8fc2dc-openshift-default-5bf4cbccc7-8hz5w"]
Jul 16 22:20:13.159736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:13.159702    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" path="/var/lib/kubelet/pods/4a366b4d-a4f3-4a75-bb2f-23302e3be8a7/volumes"
Jul 16 22:20:18.853732 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.853703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854015    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854025    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854038    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854044    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854054    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:18.854072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854060    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9471607b-a6e5-4b43-acbd-d5a5c1274096" containerName="revoke-keys"
Jul 16 22:20:18.854294 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854116    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.854294 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.854122    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4a366b4d-a4f3-4a75-bb2f-23302e3be8a7" containerName="istio-proxy"
Jul 16 22:20:18.858421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.858402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.861520 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.861495    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-4d06d9c2-gw-openshift-default-dockercfg-4b2gn\""
Jul 16 22:20:18.872378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.872358    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:20:18.900885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.900860    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.900980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.900895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.900980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.900922    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcsch\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.900980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.900976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.901081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.901038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.901081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.901059    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.901140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.901086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.901140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.901115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:18.901200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:18.901139    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002344 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002310    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002344 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002349    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcsch\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002422    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002522    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002552    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002597    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.002954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.003017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.002956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.003077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.003055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.003077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.003067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.003170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.003151    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.004854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.004831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.005062 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.005046    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.017015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.016988    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcsch\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.023083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.023056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token\") pod \"e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") " pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.168454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.168374    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:19.309821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.309798    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:20:19.311695 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:20:19.311671    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6898338_9eaa_4a76_a074_134bbb8fd13b.slice/crio-1ae6d8c9c126acac9eba935794e1187c1416692609c391c7b285fa33ebe7d21c WatchSource:0}: Error finding container 1ae6d8c9c126acac9eba935794e1187c1416692609c391c7b285fa33ebe7d21c: Status 404 returned error can't find the container with id 1ae6d8c9c126acac9eba935794e1187c1416692609c391c7b285fa33ebe7d21c
Jul 16 22:20:19.313642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.313616    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:19.313692 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.313680    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:19.313731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:19.313705    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:20.273038 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:20.272989    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerStarted","Data":"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"}
Jul 16 22:20:20.273418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:20.273057    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerStarted","Data":"1ae6d8c9c126acac9eba935794e1187c1416692609c391c7b285fa33ebe7d21c"}
Jul 16 22:20:20.301222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:20.301177    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podStartSLOduration=2.3011620280000002 podStartE2EDuration="2.301162028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:20.300164249 +0000 UTC m=+2007.629604152" watchObservedRunningTime="2026-07-16 22:20:20.301162028 +0000 UTC m=+2007.630601929"
Jul 16 22:20:21.168556 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:21.168527    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:21.174166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:21.174142    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:21.278029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:21.277991    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:21.279123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:21.279099    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:23.601579 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:23.601544    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:20:24.155465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.155436    2581 scope.go:117] "RemoveContainer" containerID="d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92"
Jul 16 22:20:24.658962 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.658900    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:24.662999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.662975    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:24.671212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.671170    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:24.758256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.756770    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv\" (UID: \"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:24.858188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.858136    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv\" (UID: \"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:24.859480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.859448    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv\" (UID: \"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:24.978450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:24.978361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:25.123800 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.123655    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:25.125974 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:20:25.125948    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ebaa0c2_4ce8_4c79_9252_11ae5ef44c29.slice/crio-12107abb304169dd0be9b8a18462db0418eb9716549e50d340dc83729c576409 WatchSource:0}: Error finding container 12107abb304169dd0be9b8a18462db0418eb9716549e50d340dc83729c576409: Status 404 returned error can't find the container with id 12107abb304169dd0be9b8a18462db0418eb9716549e50d340dc83729c576409
Jul 16 22:20:25.297625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.297541    2581 generic.go:356] "Generic (PLEG): container finished" podID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerID="03ad16e98065aa73ee056d73ebd1ee8fa46517052440764b6bd1969c884ff639" exitCode=6
Jul 16 22:20:25.297771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.297622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerDied","Data":"03ad16e98065aa73ee056d73ebd1ee8fa46517052440764b6bd1969c884ff639"}
Jul 16 22:20:25.297771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.297677    2581 scope.go:117] "RemoveContainer" containerID="d6849d7407ce378527c4fd954b402b0cf15a78da78ab1fdf3e00c3f6a3274b92"
Jul 16 22:20:25.297962 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.297946    2581 scope.go:117] "RemoveContainer" containerID="03ad16e98065aa73ee056d73ebd1ee8fa46517052440764b6bd1969c884ff639"
Jul 16 22:20:25.298187 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:25.298162    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw_odh-ai-gateway-infra(96a728ea-775e-4085-ac1c-2d98f1940c1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a"
Jul 16 22:20:25.298980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:25.298958    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" event={"ID":"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29","Type":"ContainerStarted","Data":"12107abb304169dd0be9b8a18462db0418eb9716549e50d340dc83729c576409"}
Jul 16 22:20:26.304918 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.304884    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerID="4db2e06b5b2ce4ea7a040096ee9951c67b260cf6dbb99a6f7dcd555981c03b57" exitCode=7
Jul 16 22:20:26.305367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.304978    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" event={"ID":"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29","Type":"ContainerDied","Data":"4db2e06b5b2ce4ea7a040096ee9951c67b260cf6dbb99a6f7dcd555981c03b57"}
Jul 16 22:20:26.305367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.305166    2581 scope.go:117] "RemoveContainer" containerID="4db2e06b5b2ce4ea7a040096ee9951c67b260cf6dbb99a6f7dcd555981c03b57"
Jul 16 22:20:26.330907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.327095    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:26.495395 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.495372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:26.583149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.583092    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca\") pod \"96a728ea-775e-4085-ac1c-2d98f1940c1a\" (UID: \"96a728ea-775e-4085-ac1c-2d98f1940c1a\") "
Jul 16 22:20:26.583448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.583424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96a728ea-775e-4085-ac1c-2d98f1940c1a" (UID: "96a728ea-775e-4085-ac1c-2d98f1940c1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:26.684334 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:26.684301    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96a728ea-775e-4085-ac1c-2d98f1940c1a-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:27.312605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.312568    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerID="ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29" exitCode=7
Jul 16 22:20:27.313091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.312655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" event={"ID":"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29","Type":"ContainerDied","Data":"ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29"}
Jul 16 22:20:27.313091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.312718    2581 scope.go:117] "RemoveContainer" containerID="4db2e06b5b2ce4ea7a040096ee9951c67b260cf6dbb99a6f7dcd555981c03b57"
Jul 16 22:20:27.313091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.312923    2581 scope.go:117] "RemoveContainer" containerID="ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29"
Jul 16 22:20:27.313222 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:27.313166    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv_odh-ai-gateway-infra(8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29"
Jul 16 22:20:27.314097 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.314078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw" event={"ID":"96a728ea-775e-4085-ac1c-2d98f1940c1a","Type":"ContainerDied","Data":"59ed5f30541ea19a6b13188688b536f43ababa5a1665d4978066414a48743ad2"}
Jul 16 22:20:27.314164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.314098    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"
Jul 16 22:20:27.323509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.323492    2581 scope.go:117] "RemoveContainer" containerID="03ad16e98065aa73ee056d73ebd1ee8fa46517052440764b6bd1969c884ff639"
Jul 16 22:20:27.351298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.351264    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:27.354015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:27.353989    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-wlclw"]
Jul 16 22:20:28.321442 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:28.321411    2581 scope.go:117] "RemoveContainer" containerID="ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29"
Jul 16 22:20:28.321912 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:28.321630    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv_odh-ai-gateway-infra(8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29"
Jul 16 22:20:29.161578 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:29.161542    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" path="/var/lib/kubelet/pods/96a728ea-775e-4085-ac1c-2d98f1940c1a/volumes"
Jul 16 22:20:31.335179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.335154    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerID="91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221" exitCode=0
Jul 16 22:20:31.335545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.335254    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerDied","Data":"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"}
Jul 16 22:20:31.335616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.335601    2581 scope.go:117] "RemoveContainer" containerID="91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"
Jul 16 22:20:31.336528 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.336505    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:31.336582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.336573    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:31.336616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:31.336605    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:20:32.343295 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:32.343247    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerStarted","Data":"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"}
Jul 16 22:20:33.169120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:33.169087    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:33.170484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:33.170457    2581 patch_prober.go:28] interesting pod/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 16 22:20:33.170628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:33.170531    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 16 22:20:34.169723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:34.169678    2581 patch_prober.go:28] interesting pod/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 16 22:20:34.170130 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:34.169753    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 16 22:20:35.169504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:35.169471    2581 patch_prober.go:28] interesting pod/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 16 22:20:35.169682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:35.169538    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 16 22:20:36.168783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:36.168739    2581 patch_prober.go:28] interesting pod/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused" start-of-body=
Jul 16 22:20:36.169173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:36.168805    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.33:15021/healthz/ready\": dial tcp 10.133.0.33:15021: connect: connection refused"
Jul 16 22:20:36.278825 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:36.278793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:37.172859 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:37.172830    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:37.173802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:37.173785    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:20:41.154872 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:41.154834    2581 scope.go:117] "RemoveContainer" containerID="ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29"
Jul 16 22:20:42.399204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:42.399164    2581 generic.go:356] "Generic (PLEG): container finished" podID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerID="bfde52be300739881840802188da1a1c9a45aef6c7cd3ec41466db889995d4bb" exitCode=7
Jul 16 22:20:42.399699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:42.399264    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" event={"ID":"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29","Type":"ContainerDied","Data":"bfde52be300739881840802188da1a1c9a45aef6c7cd3ec41466db889995d4bb"}
Jul 16 22:20:42.399699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:42.399341    2581 scope.go:117] "RemoveContainer" containerID="ce61ecb50693bbd57ff555df256868adfa51c3198ce0fd6e1618d8c251664a29"
Jul 16 22:20:42.399699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:42.399688    2581 scope.go:117] "RemoveContainer" containerID="bfde52be300739881840802188da1a1c9a45aef6c7cd3ec41466db889995d4bb"
Jul 16 22:20:42.399948 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:20:42.399923    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv_odh-ai-gateway-infra(8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29"
Jul 16 22:20:43.428121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:43.428090    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:43.563456 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:43.563430    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:43.739032 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:43.738940    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca\") pod \"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29\" (UID: \"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29\") "
Jul 16 22:20:43.739321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:43.739300    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" (UID: "8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:43.839884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:43.839844    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:20:44.410536 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:44.410503    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"
Jul 16 22:20:44.410536 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:44.410507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv" event={"ID":"8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29","Type":"ContainerDied","Data":"12107abb304169dd0be9b8a18462db0418eb9716549e50d340dc83729c576409"}
Jul 16 22:20:44.410790 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:44.410578    2581 scope.go:117] "RemoveContainer" containerID="bfde52be300739881840802188da1a1c9a45aef6c7cd3ec41466db889995d4bb"
Jul 16 22:20:44.431275 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:44.431248    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:44.438323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:44.438297    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-26qgv"]
Jul 16 22:20:45.160680 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:20:45.160645    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" path="/var/lib/kubelet/pods/8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29/volumes"
Jul 16 22:21:06.234633 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234583    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234914    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234925    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234941    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234946    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234954    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234960    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234970    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234975    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234983    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234988    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234993    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.234999    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235045    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235052    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235072    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235078    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ebaa0c2-4ce8-4c79-9252-11ae5ef44c29" containerName="revoke-keys"
Jul 16 22:21:06.235563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235085    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.235563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.235091    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="96a728ea-775e-4085-ac1c-2d98f1940c1a" containerName="revoke-keys"
Jul 16 22:21:06.237749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.237733    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:06.240370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.240332    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:06.240370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.240357    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:21:06.246805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.246770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:06.327119 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.327080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-85577\" (UID: \"72290db8-5e07-4eb3-af0b-992fc7635aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:06.428449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.428401    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-85577\" (UID: \"72290db8-5e07-4eb3-af0b-992fc7635aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:06.429041 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.429019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-85577\" (UID: \"72290db8-5e07-4eb3-af0b-992fc7635aba\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:06.548812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.548721    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:06.683555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:06.683527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:06.685855 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:21:06.685833    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72290db8_5e07_4eb3_af0b_992fc7635aba.slice/crio-e81f8f26d749ffd76fb84ce92e09342e680737ec9f79b66b332759435a63dfbf WatchSource:0}: Error finding container e81f8f26d749ffd76fb84ce92e09342e680737ec9f79b66b332759435a63dfbf: Status 404 returned error can't find the container with id e81f8f26d749ffd76fb84ce92e09342e680737ec9f79b66b332759435a63dfbf
Jul 16 22:21:07.518269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:07.518155    2581 generic.go:356] "Generic (PLEG): container finished" podID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerID="132f3a2297b4d0af5aca3de30a812023f869224935a70b6db6fd6222be2cf795" exitCode=6
Jul 16 22:21:07.518269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:07.518197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerDied","Data":"132f3a2297b4d0af5aca3de30a812023f869224935a70b6db6fd6222be2cf795"}
Jul 16 22:21:07.518269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:07.518258    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerStarted","Data":"e81f8f26d749ffd76fb84ce92e09342e680737ec9f79b66b332759435a63dfbf"}
Jul 16 22:21:07.518689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:07.518463    2581 scope.go:117] "RemoveContainer" containerID="132f3a2297b4d0af5aca3de30a812023f869224935a70b6db6fd6222be2cf795"
Jul 16 22:21:08.524674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:08.524582    2581 generic.go:356] "Generic (PLEG): container finished" podID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerID="b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922" exitCode=6
Jul 16 22:21:08.524674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:08.524628    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerDied","Data":"b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922"}
Jul 16 22:21:08.525209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:08.524678    2581 scope.go:117] "RemoveContainer" containerID="132f3a2297b4d0af5aca3de30a812023f869224935a70b6db6fd6222be2cf795"
Jul 16 22:21:08.525209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:08.524850    2581 scope.go:117] "RemoveContainer" containerID="b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922"
Jul 16 22:21:08.525209 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:08.525085    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-85577_odh-ai-gateway-infra(72290db8-5e07-4eb3-af0b-992fc7635aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba"
Jul 16 22:21:09.530572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:09.530544    2581 scope.go:117] "RemoveContainer" containerID="b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922"
Jul 16 22:21:09.531042 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:09.530783    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-85577_odh-ai-gateway-infra(72290db8-5e07-4eb3-af0b-992fc7635aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba"
Jul 16 22:21:14.700893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.700816    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:14.705353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.705338    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:14.711461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.711406    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:14.800318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.800290    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p\" (UID: \"e68681d0-46bb-48fc-9d47-048310888bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:14.901107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.901075    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p\" (UID: \"e68681d0-46bb-48fc-9d47-048310888bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:14.901719 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:14.901699    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p\" (UID: \"e68681d0-46bb-48fc-9d47-048310888bde\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:15.017410 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:15.017322    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:15.147718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:15.147691    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:15.149921 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:21:15.149895    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode68681d0_46bb_48fc_9d47_048310888bde.slice/crio-c651c2b7ee694180182a4dda609f0947deea6780b9ac262b0bedd83ef7e7b888 WatchSource:0}: Error finding container c651c2b7ee694180182a4dda609f0947deea6780b9ac262b0bedd83ef7e7b888: Status 404 returned error can't find the container with id c651c2b7ee694180182a4dda609f0947deea6780b9ac262b0bedd83ef7e7b888
Jul 16 22:21:15.557850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:15.557817    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerStarted","Data":"26d9778a8207d0f8203e740bfe0ea30428033b61d07479f775cf536df24a01d2"}
Jul 16 22:21:15.558001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:15.557869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerStarted","Data":"c651c2b7ee694180182a4dda609f0947deea6780b9ac262b0bedd83ef7e7b888"}
Jul 16 22:21:15.575081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:15.575031    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" podStartSLOduration=1.5750152389999998 podStartE2EDuration="1.575015239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:15.573514992 +0000 UTC m=+2062.902954910" watchObservedRunningTime="2026-07-16 22:21:15.575015239 +0000 UTC m=+2062.904455158"
Jul 16 22:21:16.563100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:16.563061    2581 generic.go:356] "Generic (PLEG): container finished" podID="e68681d0-46bb-48fc-9d47-048310888bde" containerID="26d9778a8207d0f8203e740bfe0ea30428033b61d07479f775cf536df24a01d2" exitCode=7
Jul 16 22:21:16.563532 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:16.563115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerDied","Data":"26d9778a8207d0f8203e740bfe0ea30428033b61d07479f775cf536df24a01d2"}
Jul 16 22:21:16.563532 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:16.563387    2581 scope.go:117] "RemoveContainer" containerID="26d9778a8207d0f8203e740bfe0ea30428033b61d07479f775cf536df24a01d2"
Jul 16 22:21:17.569473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:17.569380    2581 generic.go:356] "Generic (PLEG): container finished" podID="e68681d0-46bb-48fc-9d47-048310888bde" containerID="cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977" exitCode=7
Jul 16 22:21:17.569473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:17.569451    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerDied","Data":"cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977"}
Jul 16 22:21:17.569974 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:17.569509    2581 scope.go:117] "RemoveContainer" containerID="26d9778a8207d0f8203e740bfe0ea30428033b61d07479f775cf536df24a01d2"
Jul 16 22:21:17.569974 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:17.569695    2581 scope.go:117] "RemoveContainer" containerID="cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977"
Jul 16 22:21:17.569974 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:17.569912    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p_odh-ai-gateway-infra(e68681d0-46bb-48fc-9d47-048310888bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" podUID="e68681d0-46bb-48fc-9d47-048310888bde"
Jul 16 22:21:18.575384 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:18.575345    2581 scope.go:117] "RemoveContainer" containerID="cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977"
Jul 16 22:21:18.575855 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:18.575585    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p_odh-ai-gateway-infra(e68681d0-46bb-48fc-9d47-048310888bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" podUID="e68681d0-46bb-48fc-9d47-048310888bde"
Jul 16 22:21:22.154688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:22.154653    2581 scope.go:117] "RemoveContainer" containerID="b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922"
Jul 16 22:21:22.593662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:22.593629    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerStarted","Data":"529b074a82b78ad60b293f35ef237c3a83d543e66840d3f0b7b7558b7da59b79"}
Jul 16 22:21:22.610049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:22.610007    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" podStartSLOduration=16.609992833 podStartE2EDuration="16.609992833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:22.609468423 +0000 UTC m=+2069.938908325" watchObservedRunningTime="2026-07-16 22:21:22.609992833 +0000 UTC m=+2069.939432742"
Jul 16 22:21:23.599928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:23.599893    2581 generic.go:356] "Generic (PLEG): container finished" podID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerID="529b074a82b78ad60b293f35ef237c3a83d543e66840d3f0b7b7558b7da59b79" exitCode=6
Jul 16 22:21:23.600362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:23.599969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerDied","Data":"529b074a82b78ad60b293f35ef237c3a83d543e66840d3f0b7b7558b7da59b79"}
Jul 16 22:21:23.600362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:23.600032    2581 scope.go:117] "RemoveContainer" containerID="b8643091a72e181ce70b038b805c340d119afd612afa31ddf14ac2a782c50922"
Jul 16 22:21:23.600457 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:23.600365    2581 scope.go:117] "RemoveContainer" containerID="529b074a82b78ad60b293f35ef237c3a83d543e66840d3f0b7b7558b7da59b79"
Jul 16 22:21:23.600583 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:23.600562    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-85577_odh-ai-gateway-infra(72290db8-5e07-4eb3-af0b-992fc7635aba)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba"
Jul 16 22:21:23.626061 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:23.626027    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:24.737891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.737867    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:24.790560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.790522    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:21:24.790776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.790751    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy" containerID="cri-o://c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552" gracePeriod=30
Jul 16 22:21:24.889336 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.889249    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca\") pod \"72290db8-5e07-4eb3-af0b-992fc7635aba\" (UID: \"72290db8-5e07-4eb3-af0b-992fc7635aba\") "
Jul 16 22:21:24.889619 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.889595    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "72290db8-5e07-4eb3-af0b-992fc7635aba" (UID: "72290db8-5e07-4eb3-af0b-992fc7635aba"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:24.990454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:24.990409    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/72290db8-5e07-4eb3-af0b-992fc7635aba-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:25.612704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:25.612663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577" event={"ID":"72290db8-5e07-4eb3-af0b-992fc7635aba","Type":"ContainerDied","Data":"e81f8f26d749ffd76fb84ce92e09342e680737ec9f79b66b332759435a63dfbf"}
Jul 16 22:21:25.612704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:25.612693    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"
Jul 16 22:21:25.612986 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:25.612736    2581 scope.go:117] "RemoveContainer" containerID="529b074a82b78ad60b293f35ef237c3a83d543e66840d3f0b7b7558b7da59b79"
Jul 16 22:21:25.647674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:25.647645    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:25.650028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:25.650002    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-85577"]
Jul 16 22:21:27.161138 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:27.161107    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" path="/var/lib/kubelet/pods/72290db8-5e07-4eb3-af0b-992fc7635aba/volumes"
Jul 16 22:21:27.511872 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:27.511787    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:21:30.049496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.049474    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:21:30.237496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237459    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237511    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237583    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237611    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237654    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237679    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237708    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcsch\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.237980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237774    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:30.237980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237833    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:30.237980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.237888    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo\") pod \"b6898338-9eaa-4a76-a074-134bbb8fd13b\" (UID: \"b6898338-9eaa-4a76-a074-134bbb8fd13b\") "
Jul 16 22:21:30.238188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:30.238188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:30.238329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238192    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b6898338-9eaa-4a76-a074-134bbb8fd13b-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.238329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238207    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.238329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238221    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.238329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238257    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.238686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.238434    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data" (OuterVolumeSpecName: "istio-data") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:30.240172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.240143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch" (OuterVolumeSpecName: "kube-api-access-fcsch") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "kube-api-access-fcsch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:30.240172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.240148    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:21:30.240406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.240371    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token" (OuterVolumeSpecName: "istio-token") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:21:30.240504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.240482    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "b6898338-9eaa-4a76-a074-134bbb8fd13b" (UID: "b6898338-9eaa-4a76-a074-134bbb8fd13b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:21:30.339001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.338958    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.339001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.338992    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.339001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.339004    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.339259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.339016    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcsch\" (UniqueName: \"kubernetes.io/projected/b6898338-9eaa-4a76-a074-134bbb8fd13b-kube-api-access-fcsch\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.339259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.339028    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b6898338-9eaa-4a76-a074-134bbb8fd13b-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:30.636695 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.636658    2581 generic.go:356] "Generic (PLEG): container finished" podID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerID="c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552" exitCode=0
Jul 16 22:21:30.636899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.636715    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"
Jul 16 22:21:30.636899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.636718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerDied","Data":"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"}
Jul 16 22:21:30.636899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.636775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c" event={"ID":"b6898338-9eaa-4a76-a074-134bbb8fd13b","Type":"ContainerDied","Data":"1ae6d8c9c126acac9eba935794e1187c1416692609c391c7b285fa33ebe7d21c"}
Jul 16 22:21:30.636899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.636805    2581 scope.go:117] "RemoveContainer" containerID="c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"
Jul 16 22:21:30.646108 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.646088    2581 scope.go:117] "RemoveContainer" containerID="91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"
Jul 16 22:21:30.654117 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.654099    2581 scope.go:117] "RemoveContainer" containerID="c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"
Jul 16 22:21:30.654398 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:30.654376    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552\": container with ID starting with c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552 not found: ID does not exist" containerID="c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"
Jul 16 22:21:30.654458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.654408    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552"} err="failed to get container status \"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552\": rpc error: code = NotFound desc = could not find container \"c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552\": container with ID starting with c8fbac94fe1e0679000ceb6ef907a2d9db9f5b74e76ee1cdb03e8312f06e3552 not found: ID does not exist"
Jul 16 22:21:30.654458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.654426    2581 scope.go:117] "RemoveContainer" containerID="91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"
Jul 16 22:21:30.654676 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:30.654662    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221\": container with ID starting with 91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221 not found: ID does not exist" containerID="91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"
Jul 16 22:21:30.654713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.654681    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221"} err="failed to get container status \"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221\": rpc error: code = NotFound desc = could not find container \"91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221\": container with ID starting with 91a73c5822010d4ac4a0fc8aeda112c1019c8134da0eaa5e045b420927e42221 not found: ID does not exist"
Jul 16 22:21:30.679341 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.679316    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:21:30.695375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:30.695346    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-4d06d9c2-gw-openshift-default-75c97496c9-tnl5c"]
Jul 16 22:21:31.161062 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:31.161025    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" path="/var/lib/kubelet/pods/b6898338-9eaa-4a76-a074-134bbb8fd13b/volumes"
Jul 16 22:21:32.154799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:32.154769    2581 scope.go:117] "RemoveContainer" containerID="cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977"
Jul 16 22:21:32.649832 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:32.649790    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerStarted","Data":"d50c643de89a1729c08ba27ad9816a85eced0d613af795eacc0b2640fa1ce264"}
Jul 16 22:21:33.655944 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:33.655910    2581 generic.go:356] "Generic (PLEG): container finished" podID="e68681d0-46bb-48fc-9d47-048310888bde" containerID="d50c643de89a1729c08ba27ad9816a85eced0d613af795eacc0b2640fa1ce264" exitCode=6
Jul 16 22:21:33.656452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:33.655984    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerDied","Data":"d50c643de89a1729c08ba27ad9816a85eced0d613af795eacc0b2640fa1ce264"}
Jul 16 22:21:33.656452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:33.656062    2581 scope.go:117] "RemoveContainer" containerID="cdfe629ea6937b0a82ba440b35f72a49ea1319c0f8c817f85aad2af41ffbf977"
Jul 16 22:21:33.656452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:33.656378    2581 scope.go:117] "RemoveContainer" containerID="d50c643de89a1729c08ba27ad9816a85eced0d613af795eacc0b2640fa1ce264"
Jul 16 22:21:33.656611 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:33.656589    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p_odh-ai-gateway-infra(e68681d0-46bb-48fc-9d47-048310888bde)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" podUID="e68681d0-46bb-48fc-9d47-048310888bde"
Jul 16 22:21:33.691928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:33.691900    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:34.801390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:34.801366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:34.877183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:34.877140    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca\") pod \"e68681d0-46bb-48fc-9d47-048310888bde\" (UID: \"e68681d0-46bb-48fc-9d47-048310888bde\") "
Jul 16 22:21:34.877577 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:34.877551    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e68681d0-46bb-48fc-9d47-048310888bde" (UID: "e68681d0-46bb-48fc-9d47-048310888bde"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:34.977778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:34.977691    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e68681d0-46bb-48fc-9d47-048310888bde-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:21:35.667149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:35.667117    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"
Jul 16 22:21:35.667374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:35.667158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p" event={"ID":"e68681d0-46bb-48fc-9d47-048310888bde","Type":"ContainerDied","Data":"c651c2b7ee694180182a4dda609f0947deea6780b9ac262b0bedd83ef7e7b888"}
Jul 16 22:21:35.667374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:35.667223    2581 scope.go:117] "RemoveContainer" containerID="d50c643de89a1729c08ba27ad9816a85eced0d613af795eacc0b2640fa1ce264"
Jul 16 22:21:35.688682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:35.688659    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:35.695320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:35.695297    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-59x4p"]
Jul 16 22:21:37.159817 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.159778    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68681d0-46bb-48fc-9d47-048310888bde" path="/var/lib/kubelet/pods/e68681d0-46bb-48fc-9d47-048310888bde/volumes"
Jul 16 22:21:37.603996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.603961    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:21:37.604478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604456    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.604478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604479    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604488    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604494    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604504    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604509    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604521    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604526    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604531    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604537    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604547    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604552    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604606    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:37.604625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604625    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.605117 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.604634    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:37.608999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.608982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.611527 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.611505    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-24cdbda3-openshift-default-dockercfg-l74vw\""
Jul 16 22:21:37.626617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.626576    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:21:37.693640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693609    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693664    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693757    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693788    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bz7rh\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693807    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.693857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693834    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.694074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693895    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.694074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.693938    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795156 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795116    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795156 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795159    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bz7rh\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795319    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795353    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795398    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795426    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795615    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.795938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.795900    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.796140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.796118    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.797822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.797801    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.797934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.797916    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.803668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.803637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.803776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.803726    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bz7rh\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh\") pod \"e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") " pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:37.920134 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:37.920059    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:38.053408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.053383    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:21:38.055940 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:21:38.055912    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99b84cbf_5d43_47b4_bc12_67d8f63e4aa5.slice/crio-9fb7edd239e067be797270590e2901f7578213233b9322f8076efed14b491465 WatchSource:0}: Error finding container 9fb7edd239e067be797270590e2901f7578213233b9322f8076efed14b491465: Status 404 returned error can't find the container with id 9fb7edd239e067be797270590e2901f7578213233b9322f8076efed14b491465
Jul 16 22:21:38.058164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.058135    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:38.058256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.058211    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:38.058301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.058266    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:38.683249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.683185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerStarted","Data":"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"}
Jul 16 22:21:38.683658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.683302    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerStarted","Data":"9fb7edd239e067be797270590e2901f7578213233b9322f8076efed14b491465"}
Jul 16 22:21:38.707625 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.707581    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podStartSLOduration=1.707567128 podStartE2EDuration="1.707567128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:38.705826527 +0000 UTC m=+2086.035266431" watchObservedRunningTime="2026-07-16 22:21:38.707567128 +0000 UTC m=+2086.037007030"
Jul 16 22:21:38.921090 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:38.921054    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:39.924936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:39.924909    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:40.692318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:40.692286    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:40.693290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:40.693272    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:41.314882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:41.314801    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:21:43.869361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869330    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:21:43.869754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869702    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:43.869754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869715    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:43.869754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869727    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:43.869754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869733    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:43.869885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869791    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="72290db8-5e07-4eb3-af0b-992fc7635aba" containerName="revoke-keys"
Jul 16 22:21:43.869885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869801    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68681d0-46bb-48fc-9d47-048310888bde" containerName="revoke-keys"
Jul 16 22:21:43.869885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.869815    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b6898338-9eaa-4a76-a074-134bbb8fd13b" containerName="istio-proxy"
Jul 16 22:21:43.872987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.872960    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:21:43.875423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.875401    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:43.875537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.875499    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:21:43.887209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.887182    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:21:43.958749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:43.958703    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f\" (UID: \"3331ed79-32e0-4d44-9620-47c78b0bd46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:21:44.060177 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.060131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f\" (UID: \"3331ed79-32e0-4d44-9620-47c78b0bd46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:21:44.061383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.061185    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f\" (UID: \"3331ed79-32e0-4d44-9620-47c78b0bd46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:21:44.188827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.188739    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:21:44.363086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.363050    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:21:44.369457 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:21:44.369413    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3331ed79_32e0_4d44_9620_47c78b0bd46e.slice/crio-d363d47182463618548f18bae276e2d12c44e2d2ef6c8b331fb815d2fcb1c6e3 WatchSource:0}: Error finding container d363d47182463618548f18bae276e2d12c44e2d2ef6c8b331fb815d2fcb1c6e3: Status 404 returned error can't find the container with id d363d47182463618548f18bae276e2d12c44e2d2ef6c8b331fb815d2fcb1c6e3
Jul 16 22:21:44.710907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.710871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerStarted","Data":"a409b2463920a7383391e89fc6a527102aa01819138ae42c3ba80af144de7b96"}
Jul 16 22:21:44.711053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.710931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerStarted","Data":"d363d47182463618548f18bae276e2d12c44e2d2ef6c8b331fb815d2fcb1c6e3"}
Jul 16 22:21:44.727644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:44.727599    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" podStartSLOduration=1.727583469 podStartE2EDuration="1.727583469s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:44.726345939 +0000 UTC m=+2092.055785842" watchObservedRunningTime="2026-07-16 22:21:44.727583469 +0000 UTC m=+2092.057023366"
Jul 16 22:21:45.716779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:45.716742    2581 generic.go:356] "Generic (PLEG): container finished" podID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerID="a409b2463920a7383391e89fc6a527102aa01819138ae42c3ba80af144de7b96" exitCode=7
Jul 16 22:21:45.717263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:45.716837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerDied","Data":"a409b2463920a7383391e89fc6a527102aa01819138ae42c3ba80af144de7b96"}
Jul 16 22:21:45.717263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:45.716972    2581 scope.go:117] "RemoveContainer" containerID="a409b2463920a7383391e89fc6a527102aa01819138ae42c3ba80af144de7b96"
Jul 16 22:21:46.723103 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:46.723064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerStarted","Data":"4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"}
Jul 16 22:21:47.730300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:47.730262    2581 generic.go:356] "Generic (PLEG): container finished" podID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerID="4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1" exitCode=7
Jul 16 22:21:47.730779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:47.730341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerDied","Data":"4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"}
Jul 16 22:21:47.730779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:47.730387    2581 scope.go:117] "RemoveContainer" containerID="a409b2463920a7383391e89fc6a527102aa01819138ae42c3ba80af144de7b96"
Jul 16 22:21:47.730779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:47.730626    2581 scope.go:117] "RemoveContainer" containerID="4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"
Jul 16 22:21:47.730904 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:47.730882    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f_odh-ai-gateway-infra(3331ed79-32e0-4d44-9620-47c78b0bd46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e"
Jul 16 22:21:48.736312 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:48.736281    2581 scope.go:117] "RemoveContainer" containerID="4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"
Jul 16 22:21:48.736685 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:48.736559    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f_odh-ai-gateway-infra(3331ed79-32e0-4d44-9620-47c78b0bd46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e"
Jul 16 22:21:49.741566 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.741479    2581 generic.go:356] "Generic (PLEG): container finished" podID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerID="a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036" exitCode=0
Jul 16 22:21:49.741972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.741561    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerDied","Data":"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"}
Jul 16 22:21:49.741972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.741934    2581 scope.go:117] "RemoveContainer" containerID="a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"
Jul 16 22:21:49.742923 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.742896    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:49.743029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.742974    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:49.743094 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:49.743025    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:21:50.750939 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:50.750883    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerStarted","Data":"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"}
Jul 16 22:21:50.920960 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:50.920919    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:50.922065 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:50.922042    2581 patch_prober.go:28] interesting pod/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 22:21:50.922144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:50.922096    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 22:21:51.921525 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:51.921477    2581 patch_prober.go:28] interesting pod/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 22:21:51.921919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:51.921547    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 22:21:52.920504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:52.920466    2581 patch_prober.go:28] interesting pod/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 22:21:52.920692 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:52.920525    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 22:21:53.920881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:53.920839    2581 patch_prober.go:28] interesting pod/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused" start-of-body=
Jul 16 22:21:53.921298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:53.920913    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.37:15021/healthz/ready\": dial tcp 10.133.0.37:15021: connect: connection refused"
Jul 16 22:21:54.440956 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:54.440928    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:21:54.468673 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:54.468640    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:21:54.924825 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:54.924793    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:54.925245 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:54.925104    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:54.925713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:54.925695    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:21:56.290652 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.290614    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:21:56.293978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.293956    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:21:56.304255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.304198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:21:56.385399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.385340    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9\" (UID: \"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:21:56.486846 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.486805    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9\" (UID: \"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:21:56.487552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.487529    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9\" (UID: \"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:21:56.604146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.604111    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:21:56.767982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.767938    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:21:56.788286 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:56.788256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" event={"ID":"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5","Type":"ContainerStarted","Data":"585c577fe3583cf39bc8442e96cdcf9447f29179a3a70ecf707e7e1dd7d49415"}
Jul 16 22:21:57.793949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:57.793912    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerID="228417cd598590efb2d9da295976bd3c6163f6dc2476514702fe2907f76194fe" exitCode=6
Jul 16 22:21:57.794361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:57.793972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" event={"ID":"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5","Type":"ContainerDied","Data":"228417cd598590efb2d9da295976bd3c6163f6dc2476514702fe2907f76194fe"}
Jul 16 22:21:57.794361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:57.794248    2581 scope.go:117] "RemoveContainer" containerID="228417cd598590efb2d9da295976bd3c6163f6dc2476514702fe2907f76194fe"
Jul 16 22:21:58.800255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:58.800136    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerID="6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79" exitCode=6
Jul 16 22:21:58.800255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:58.800216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" event={"ID":"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5","Type":"ContainerDied","Data":"6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79"}
Jul 16 22:21:58.800724 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:58.800286    2581 scope.go:117] "RemoveContainer" containerID="228417cd598590efb2d9da295976bd3c6163f6dc2476514702fe2907f76194fe"
Jul 16 22:21:58.800724 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:58.800483    2581 scope.go:117] "RemoveContainer" containerID="6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79"
Jul 16 22:21:58.800724 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:58.800706    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9_odh-ai-gateway-infra(dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5"
Jul 16 22:21:59.806330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:21:59.806303    2581 scope.go:117] "RemoveContainer" containerID="6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79"
Jul 16 22:21:59.806718 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:21:59.806475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9_odh-ai-gateway-infra(dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5"
Jul 16 22:22:04.155244 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:04.155205    2581 scope.go:117] "RemoveContainer" containerID="4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"
Jul 16 22:22:04.829992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:04.829956    2581 generic.go:356] "Generic (PLEG): container finished" podID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerID="cc3db87690dd7492b6d23ba1116a739e4b7024068720f72ff982b9072f599fc6" exitCode=7
Jul 16 22:22:04.830171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:04.830027    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerDied","Data":"cc3db87690dd7492b6d23ba1116a739e4b7024068720f72ff982b9072f599fc6"}
Jul 16 22:22:04.830171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:04.830096    2581 scope.go:117] "RemoveContainer" containerID="4e591971490176250e5174c24a505405a6286e07a0e1d284b39698b6761316f1"
Jul 16 22:22:04.830433 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:04.830416    2581 scope.go:117] "RemoveContainer" containerID="cc3db87690dd7492b6d23ba1116a739e4b7024068720f72ff982b9072f599fc6"
Jul 16 22:22:04.830634 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:04.830615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f_odh-ai-gateway-infra(3331ed79-32e0-4d44-9620-47c78b0bd46e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e"
Jul 16 22:22:05.857169 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:05.857126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:22:05.996035 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:05.996008    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:22:06.172522 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.172438    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca\") pod \"3331ed79-32e0-4d44-9620-47c78b0bd46e\" (UID: \"3331ed79-32e0-4d44-9620-47c78b0bd46e\") "
Jul 16 22:22:06.172785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.172765    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3331ed79-32e0-4d44-9620-47c78b0bd46e" (UID: "3331ed79-32e0-4d44-9620-47c78b0bd46e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:06.273521 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.273483    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3331ed79-32e0-4d44-9620-47c78b0bd46e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.841642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.841598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f" event={"ID":"3331ed79-32e0-4d44-9620-47c78b0bd46e","Type":"ContainerDied","Data":"d363d47182463618548f18bae276e2d12c44e2d2ef6c8b331fb815d2fcb1c6e3"}
Jul 16 22:22:06.841828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.841657    2581 scope.go:117] "RemoveContainer" containerID="cc3db87690dd7492b6d23ba1116a739e4b7024068720f72ff982b9072f599fc6"
Jul 16 22:22:06.841828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.841613    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"
Jul 16 22:22:06.865401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.865375    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:22:06.872541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:06.872519    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gkg6f"]
Jul 16 22:22:07.160585 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:07.160493    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" path="/var/lib/kubelet/pods/3331ed79-32e0-4d44-9620-47c78b0bd46e/volumes"
Jul 16 22:22:14.155408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.155362    2581 scope.go:117] "RemoveContainer" containerID="6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79"
Jul 16 22:22:14.748829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.748789    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749290    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749327    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749335    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749353    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749458    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749473    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.749605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.749482    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3331ed79-32e0-4d44-9620-47c78b0bd46e" containerName="revoke-keys"
Jul 16 22:22:14.752173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.752151    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:14.761120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.761094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:14.839069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.839038    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j\" (UID: \"fcf32442-0300-4794-a0bd-b1da9adb74c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:14.878239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.878206    2581 generic.go:356] "Generic (PLEG): container finished" podID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerID="dcd0d82e8f5a78db451280a9a57d6048f2266af8eea0a8dd9308e689c05fbfa7" exitCode=6
Jul 16 22:22:14.878356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.878280    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" event={"ID":"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5","Type":"ContainerDied","Data":"dcd0d82e8f5a78db451280a9a57d6048f2266af8eea0a8dd9308e689c05fbfa7"}
Jul 16 22:22:14.878356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.878342    2581 scope.go:117] "RemoveContainer" containerID="6ae140c355ba85fb6665316c3db7632a72e7e8d21e7486aae37641cc0b0a3f79"
Jul 16 22:22:14.878642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.878622    2581 scope.go:117] "RemoveContainer" containerID="dcd0d82e8f5a78db451280a9a57d6048f2266af8eea0a8dd9308e689c05fbfa7"
Jul 16 22:22:14.878866 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:14.878848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9_odh-ai-gateway-infra(dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5"
Jul 16 22:22:14.940483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.940452    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j\" (UID: \"fcf32442-0300-4794-a0bd-b1da9adb74c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:14.940968 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:14.940950    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j\" (UID: \"fcf32442-0300-4794-a0bd-b1da9adb74c8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:15.118978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.118952    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:15.252003 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.251969    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:15.254107 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:22:15.254076    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcf32442_0300_4794_a0bd_b1da9adb74c8.slice/crio-90271cb2c348f1ee30966eadf9010c144757936fff3f06c904a94f002562d1fb WatchSource:0}: Error finding container 90271cb2c348f1ee30966eadf9010c144757936fff3f06c904a94f002562d1fb: Status 404 returned error can't find the container with id 90271cb2c348f1ee30966eadf9010c144757936fff3f06c904a94f002562d1fb
Jul 16 22:22:15.884355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.884322    2581 generic.go:356] "Generic (PLEG): container finished" podID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerID="e5d6f147a62c60bad2f81d6cabbbf73d812cd8aba5f96131bc50ab927a67b9d9" exitCode=6
Jul 16 22:22:15.884488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.884413    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" event={"ID":"fcf32442-0300-4794-a0bd-b1da9adb74c8","Type":"ContainerDied","Data":"e5d6f147a62c60bad2f81d6cabbbf73d812cd8aba5f96131bc50ab927a67b9d9"}
Jul 16 22:22:15.884488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.884477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" event={"ID":"fcf32442-0300-4794-a0bd-b1da9adb74c8","Type":"ContainerStarted","Data":"90271cb2c348f1ee30966eadf9010c144757936fff3f06c904a94f002562d1fb"}
Jul 16 22:22:15.884602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.884583    2581 scope.go:117] "RemoveContainer" containerID="e5d6f147a62c60bad2f81d6cabbbf73d812cd8aba5f96131bc50ab927a67b9d9"
Jul 16 22:22:15.908545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:15.908517    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:22:16.040923 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.040888    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:22:16.047908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.047886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca\") pod \"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5\" (UID: \"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5\") "
Jul 16 22:22:16.048196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.048177    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" (UID: "dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:16.148798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.148773    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:22:16.892245 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.892204    2581 generic.go:356] "Generic (PLEG): container finished" podID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerID="b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95" exitCode=6
Jul 16 22:22:16.892653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.892291    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" event={"ID":"fcf32442-0300-4794-a0bd-b1da9adb74c8","Type":"ContainerDied","Data":"b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95"}
Jul 16 22:22:16.892653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.892351    2581 scope.go:117] "RemoveContainer" containerID="e5d6f147a62c60bad2f81d6cabbbf73d812cd8aba5f96131bc50ab927a67b9d9"
Jul 16 22:22:16.892653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.892521    2581 scope.go:117] "RemoveContainer" containerID="b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95"
Jul 16 22:22:16.892811 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:16.892758    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j_odh-ai-gateway-infra(fcf32442-0300-4794-a0bd-b1da9adb74c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8"
Jul 16 22:22:16.893543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.893525    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"
Jul 16 22:22:16.893607 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.893559    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9" event={"ID":"dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5","Type":"ContainerDied","Data":"585c577fe3583cf39bc8442e96cdcf9447f29179a3a70ecf707e7e1dd7d49415"}
Jul 16 22:22:16.903844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.903828    2581 scope.go:117] "RemoveContainer" containerID="dcd0d82e8f5a78db451280a9a57d6048f2266af8eea0a8dd9308e689c05fbfa7"
Jul 16 22:22:16.922474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.922451    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:22:16.927772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:16.927744    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-fl6n9"]
Jul 16 22:22:17.160074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:17.159994    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" path="/var/lib/kubelet/pods/dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5/volumes"
Jul 16 22:22:17.899318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:17.899286    2581 scope.go:117] "RemoveContainer" containerID="b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95"
Jul 16 22:22:17.899743 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:17.899534    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j_odh-ai-gateway-infra(fcf32442-0300-4794-a0bd-b1da9adb74c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8"
Jul 16 22:22:31.155123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:31.155080    2581 scope.go:117] "RemoveContainer" containerID="b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95"
Jul 16 22:22:31.966984 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:31.966949    2581 generic.go:356] "Generic (PLEG): container finished" podID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerID="432b610e39e0cb4b2d73a2cdd5c82b5039129c8abe6404db39ac22df47eed4c9" exitCode=6
Jul 16 22:22:31.967146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:31.967024    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" event={"ID":"fcf32442-0300-4794-a0bd-b1da9adb74c8","Type":"ContainerDied","Data":"432b610e39e0cb4b2d73a2cdd5c82b5039129c8abe6404db39ac22df47eed4c9"}
Jul 16 22:22:31.967146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:31.967087    2581 scope.go:117] "RemoveContainer" containerID="b8d0c00258d3029d6ef9b200183325d85ceb98905cb53ae471574181d30cdc95"
Jul 16 22:22:31.967385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:31.967364    2581 scope.go:117] "RemoveContainer" containerID="432b610e39e0cb4b2d73a2cdd5c82b5039129c8abe6404db39ac22df47eed4c9"
Jul 16 22:22:31.967614 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:31.967596    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j_odh-ai-gateway-infra(fcf32442-0300-4794-a0bd-b1da9adb74c8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8"
Jul 16 22:22:33.007072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.007041    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:33.140054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.140026    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:33.182916 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.182876    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca\") pod \"fcf32442-0300-4794-a0bd-b1da9adb74c8\" (UID: \"fcf32442-0300-4794-a0bd-b1da9adb74c8\") "
Jul 16 22:22:33.183260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.183240    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fcf32442-0300-4794-a0bd-b1da9adb74c8" (UID: "fcf32442-0300-4794-a0bd-b1da9adb74c8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:33.284101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.284022    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcf32442-0300-4794-a0bd-b1da9adb74c8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:22:33.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.978657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"
Jul 16 22:22:33.978848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.978652    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j" event={"ID":"fcf32442-0300-4794-a0bd-b1da9adb74c8","Type":"ContainerDied","Data":"90271cb2c348f1ee30966eadf9010c144757936fff3f06c904a94f002562d1fb"}
Jul 16 22:22:33.978848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:33.978791    2581 scope.go:117] "RemoveContainer" containerID="432b610e39e0cb4b2d73a2cdd5c82b5039129c8abe6404db39ac22df47eed4c9"
Jul 16 22:22:34.011521 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:34.011488    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:34.014406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:34.014385    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-96v7j"]
Jul 16 22:22:35.159903 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:35.159866    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" path="/var/lib/kubelet/pods/fcf32442-0300-4794-a0bd-b1da9adb74c8/volumes"
Jul 16 22:22:43.924328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924291    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924613    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924623    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924635    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924641    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924647    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924654    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924665    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924670    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924718    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924727    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924733    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924738    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.924735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924742    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.925116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.924750    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:43.928853 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.928836    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:22:43.931847 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.931821    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:22:43.931986 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.931859    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:43.938453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.938430    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:22:43.977137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:43.977107    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz\" (UID: \"827b6be9-dd63-4a6d-872b-e6570154b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:22:44.078487 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:44.078456    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz\" (UID: \"827b6be9-dd63-4a6d-872b-e6570154b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:22:44.079022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:44.079003    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz\" (UID: \"827b6be9-dd63-4a6d-872b-e6570154b5a9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:22:44.240243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:44.240117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:22:44.371360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:44.371320    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:22:44.379432 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:22:44.379406    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827b6be9_dd63_4a6d_872b_e6570154b5a9.slice/crio-ac8f94062024415699c9564e52b114ae5c3bd8958e03069951d2fbcc2210d321 WatchSource:0}: Error finding container ac8f94062024415699c9564e52b114ae5c3bd8958e03069951d2fbcc2210d321: Status 404 returned error can't find the container with id ac8f94062024415699c9564e52b114ae5c3bd8958e03069951d2fbcc2210d321
Jul 16 22:22:45.032012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:45.031961    2581 generic.go:356] "Generic (PLEG): container finished" podID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerID="9d54d876add17da41df455af4148c4cf24351c398094441395861068fd8e98b2" exitCode=7
Jul 16 22:22:45.032441 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:45.032080    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" event={"ID":"827b6be9-dd63-4a6d-872b-e6570154b5a9","Type":"ContainerDied","Data":"9d54d876add17da41df455af4148c4cf24351c398094441395861068fd8e98b2"}
Jul 16 22:22:45.032441 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:45.032137    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" event={"ID":"827b6be9-dd63-4a6d-872b-e6570154b5a9","Type":"ContainerStarted","Data":"ac8f94062024415699c9564e52b114ae5c3bd8958e03069951d2fbcc2210d321"}
Jul 16 22:22:45.032441 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:45.032210    2581 scope.go:117] "RemoveContainer" containerID="9d54d876add17da41df455af4148c4cf24351c398094441395861068fd8e98b2"
Jul 16 22:22:46.038074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:46.038041    2581 generic.go:356] "Generic (PLEG): container finished" podID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerID="5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2" exitCode=7
Jul 16 22:22:46.038543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:46.038105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" event={"ID":"827b6be9-dd63-4a6d-872b-e6570154b5a9","Type":"ContainerDied","Data":"5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2"}
Jul 16 22:22:46.038543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:46.038181    2581 scope.go:117] "RemoveContainer" containerID="9d54d876add17da41df455af4148c4cf24351c398094441395861068fd8e98b2"
Jul 16 22:22:46.038543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:46.038403    2581 scope.go:117] "RemoveContainer" containerID="5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2"
Jul 16 22:22:46.038721 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:46.038619    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz_odh-ai-gateway-infra(827b6be9-dd63-4a6d-872b-e6570154b5a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9"
Jul 16 22:22:47.045765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:47.045736    2581 scope.go:117] "RemoveContainer" containerID="5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2"
Jul 16 22:22:47.046369 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:47.045984    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz_odh-ai-gateway-infra(827b6be9-dd63-4a6d-872b-e6570154b5a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9"
Jul 16 22:22:56.341494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.341457    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:22:56.341873 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.341809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:56.341873 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.341820    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcf32442-0300-4794-a0bd-b1da9adb74c8" containerName="revoke-keys"
Jul 16 22:22:56.341873 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.341831    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:56.341873 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.341837    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dfe89ba8-692f-4b6d-a0d6-fae7c0a626c5" containerName="revoke-keys"
Jul 16 22:22:56.346192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.346176    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:22:56.351750 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.351720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:22:56.487433 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.487392    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn\" (UID: \"5152ba4c-3a52-4e1b-b804-3450be12535b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:22:56.588158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.588110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn\" (UID: \"5152ba4c-3a52-4e1b-b804-3450be12535b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:22:56.588769 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.588749    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn\" (UID: \"5152ba4c-3a52-4e1b-b804-3450be12535b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:22:56.658199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.658119    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:22:56.796392 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:56.796354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:22:56.799924 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:22:56.799892    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5152ba4c_3a52_4e1b_b804_3450be12535b.slice/crio-59d5846690d5867cfc6a38ea6aa78ea67dcdfc7d12f5ac0ff74ed4b83bf926ed WatchSource:0}: Error finding container 59d5846690d5867cfc6a38ea6aa78ea67dcdfc7d12f5ac0ff74ed4b83bf926ed: Status 404 returned error can't find the container with id 59d5846690d5867cfc6a38ea6aa78ea67dcdfc7d12f5ac0ff74ed4b83bf926ed
Jul 16 22:22:57.090027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:57.089997    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerStarted","Data":"8d187c95e2f93e5648110bfd3bdc6a512a3490fb70404611bb3b40e82bd61ae4"}
Jul 16 22:22:57.090201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:57.090047    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerStarted","Data":"59d5846690d5867cfc6a38ea6aa78ea67dcdfc7d12f5ac0ff74ed4b83bf926ed"}
Jul 16 22:22:57.109954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:57.109913    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" podStartSLOduration=1.109899211 podStartE2EDuration="1.109899211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:57.10840584 +0000 UTC m=+2164.437845744" watchObservedRunningTime="2026-07-16 22:22:57.109899211 +0000 UTC m=+2164.439339115"
Jul 16 22:22:58.096424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:58.096388    2581 generic.go:356] "Generic (PLEG): container finished" podID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerID="8d187c95e2f93e5648110bfd3bdc6a512a3490fb70404611bb3b40e82bd61ae4" exitCode=6
Jul 16 22:22:58.096878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:58.096482    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerDied","Data":"8d187c95e2f93e5648110bfd3bdc6a512a3490fb70404611bb3b40e82bd61ae4"}
Jul 16 22:22:58.096878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:58.096633    2581 scope.go:117] "RemoveContainer" containerID="8d187c95e2f93e5648110bfd3bdc6a512a3490fb70404611bb3b40e82bd61ae4"
Jul 16 22:22:59.103001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:59.102966    2581 generic.go:356] "Generic (PLEG): container finished" podID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerID="f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364" exitCode=6
Jul 16 22:22:59.103494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:59.103053    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerDied","Data":"f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364"}
Jul 16 22:22:59.103494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:59.103116    2581 scope.go:117] "RemoveContainer" containerID="8d187c95e2f93e5648110bfd3bdc6a512a3490fb70404611bb3b40e82bd61ae4"
Jul 16 22:22:59.103494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:59.103395    2581 scope.go:117] "RemoveContainer" containerID="f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364"
Jul 16 22:22:59.103663 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:22:59.103617    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn_odh-ai-gateway-infra(5152ba4c-3a52-4e1b-b804-3450be12535b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b"
Jul 16 22:22:59.155273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:22:59.155248    2581 scope.go:117] "RemoveContainer" containerID="5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2"
Jul 16 22:23:00.109523 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:00.109493    2581 scope.go:117] "RemoveContainer" containerID="f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364"
Jul 16 22:23:00.109980 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:00.109725    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn_odh-ai-gateway-infra(5152ba4c-3a52-4e1b-b804-3450be12535b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b"
Jul 16 22:23:00.111054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:00.111031    2581 generic.go:356] "Generic (PLEG): container finished" podID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerID="e8784b9ed741f06850f6f7ef0c7c516b90cb70494ed951668beb7d27260aa93c" exitCode=7
Jul 16 22:23:00.111176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:00.111099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" event={"ID":"827b6be9-dd63-4a6d-872b-e6570154b5a9","Type":"ContainerDied","Data":"e8784b9ed741f06850f6f7ef0c7c516b90cb70494ed951668beb7d27260aa93c"}
Jul 16 22:23:00.111176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:00.111155    2581 scope.go:117] "RemoveContainer" containerID="5876466da133243076f4b939031e3900711e92159dad994b8d6db0f3bdbb3bd2"
Jul 16 22:23:00.111432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:00.111418    2581 scope.go:117] "RemoveContainer" containerID="e8784b9ed741f06850f6f7ef0c7c516b90cb70494ed951668beb7d27260aa93c"
Jul 16 22:23:00.111598 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:00.111580    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz_odh-ai-gateway-infra(827b6be9-dd63-4a6d-872b-e6570154b5a9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9"
Jul 16 22:23:01.150581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:01.150553    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:23:01.281834 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:01.281812    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:23:01.427444 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:01.427356    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca\") pod \"827b6be9-dd63-4a6d-872b-e6570154b5a9\" (UID: \"827b6be9-dd63-4a6d-872b-e6570154b5a9\") "
Jul 16 22:23:01.427701 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:01.427679    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "827b6be9-dd63-4a6d-872b-e6570154b5a9" (UID: "827b6be9-dd63-4a6d-872b-e6570154b5a9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:01.528935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:01.528904    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827b6be9-dd63-4a6d-872b-e6570154b5a9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:23:02.122966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:02.122936    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"
Jul 16 22:23:02.123133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:02.122965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz" event={"ID":"827b6be9-dd63-4a6d-872b-e6570154b5a9","Type":"ContainerDied","Data":"ac8f94062024415699c9564e52b114ae5c3bd8958e03069951d2fbcc2210d321"}
Jul 16 22:23:02.123133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:02.123045    2581 scope.go:117] "RemoveContainer" containerID="e8784b9ed741f06850f6f7ef0c7c516b90cb70494ed951668beb7d27260aa93c"
Jul 16 22:23:02.147819 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:02.147789    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:23:02.154738 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:02.154717    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-k2wkz"]
Jul 16 22:23:03.160222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:03.160183    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" path="/var/lib/kubelet/pods/827b6be9-dd63-4a6d-872b-e6570154b5a9/volumes"
Jul 16 22:23:04.801943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.801910    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802243    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802257    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802268    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802343    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802350    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.802541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.802357    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:04.806439 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.806422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:04.818531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.818509    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:04.957438 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:04.957397    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn\" (UID: \"e3c13108-8ea1-4fff-855a-3cdf27187cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:05.058496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:05.058403    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn\" (UID: \"e3c13108-8ea1-4fff-855a-3cdf27187cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:05.059013 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:05.058992    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn\" (UID: \"e3c13108-8ea1-4fff-855a-3cdf27187cf9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:05.116612 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:05.116588    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:05.248555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:05.248520    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:05.251749 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:23:05.251724    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3c13108_8ea1_4fff_855a_3cdf27187cf9.slice/crio-64119384f75b821d317ebf81117077d6514a0bd8fd6f3e5ed8fa3f639f12e652 WatchSource:0}: Error finding container 64119384f75b821d317ebf81117077d6514a0bd8fd6f3e5ed8fa3f639f12e652: Status 404 returned error can't find the container with id 64119384f75b821d317ebf81117077d6514a0bd8fd6f3e5ed8fa3f639f12e652
Jul 16 22:23:06.143350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:06.143252    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerID="6625579955a876ff7ffb6ae43dd5dbf9c42be927edd3d01a26291f1183f14890" exitCode=6
Jul 16 22:23:06.143350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:06.143335    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" event={"ID":"e3c13108-8ea1-4fff-855a-3cdf27187cf9","Type":"ContainerDied","Data":"6625579955a876ff7ffb6ae43dd5dbf9c42be927edd3d01a26291f1183f14890"}
Jul 16 22:23:06.143765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:06.143407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" event={"ID":"e3c13108-8ea1-4fff-855a-3cdf27187cf9","Type":"ContainerStarted","Data":"64119384f75b821d317ebf81117077d6514a0bd8fd6f3e5ed8fa3f639f12e652"}
Jul 16 22:23:06.143765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:06.143510    2581 scope.go:117] "RemoveContainer" containerID="6625579955a876ff7ffb6ae43dd5dbf9c42be927edd3d01a26291f1183f14890"
Jul 16 22:23:07.149386 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:07.149351    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerID="a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb" exitCode=6
Jul 16 22:23:07.149854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:07.149434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" event={"ID":"e3c13108-8ea1-4fff-855a-3cdf27187cf9","Type":"ContainerDied","Data":"a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb"}
Jul 16 22:23:07.149854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:07.149485    2581 scope.go:117] "RemoveContainer" containerID="6625579955a876ff7ffb6ae43dd5dbf9c42be927edd3d01a26291f1183f14890"
Jul 16 22:23:07.149854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:07.149707    2581 scope.go:117] "RemoveContainer" containerID="a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb"
Jul 16 22:23:07.150076 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:07.149948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn_odh-ai-gateway-infra(e3c13108-8ea1-4fff-855a-3cdf27187cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9"
Jul 16 22:23:08.155708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:08.155677    2581 scope.go:117] "RemoveContainer" containerID="a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb"
Jul 16 22:23:08.156098 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:08.155887    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn_odh-ai-gateway-infra(e3c13108-8ea1-4fff-855a-3cdf27187cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9"
Jul 16 22:23:15.155608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:15.155572    2581 scope.go:117] "RemoveContainer" containerID="f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364"
Jul 16 22:23:16.192049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:16.192010    2581 generic.go:356] "Generic (PLEG): container finished" podID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerID="d188f3b96d80ea0803499f9daad90b489848001b0d14c7e58153e27bb2089fef" exitCode=6
Jul 16 22:23:16.192500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:16.192082    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerDied","Data":"d188f3b96d80ea0803499f9daad90b489848001b0d14c7e58153e27bb2089fef"}
Jul 16 22:23:16.192500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:16.192143    2581 scope.go:117] "RemoveContainer" containerID="f7d487aff441bab03b1c8257ef7eedbb411afdd3a9f60c56d0cc866cc4c2a364"
Jul 16 22:23:16.192500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:16.192413    2581 scope.go:117] "RemoveContainer" containerID="d188f3b96d80ea0803499f9daad90b489848001b0d14c7e58153e27bb2089fef"
Jul 16 22:23:16.192658 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:16.192640    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn_odh-ai-gateway-infra(5152ba4c-3a52-4e1b-b804-3450be12535b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b"
Jul 16 22:23:17.216979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:17.216944    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:23:17.348394 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:17.348373    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:23:17.367797 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:17.367755    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca\") pod \"5152ba4c-3a52-4e1b-b804-3450be12535b\" (UID: \"5152ba4c-3a52-4e1b-b804-3450be12535b\") "
Jul 16 22:23:17.368196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:17.368163    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5152ba4c-3a52-4e1b-b804-3450be12535b" (UID: "5152ba4c-3a52-4e1b-b804-3450be12535b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:17.468883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:17.468798    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5152ba4c-3a52-4e1b-b804-3450be12535b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:23:18.203320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:18.203290    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"
Jul 16 22:23:18.203967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:18.203319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn" event={"ID":"5152ba4c-3a52-4e1b-b804-3450be12535b","Type":"ContainerDied","Data":"59d5846690d5867cfc6a38ea6aa78ea67dcdfc7d12f5ac0ff74ed4b83bf926ed"}
Jul 16 22:23:18.203967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:18.203389    2581 scope.go:117] "RemoveContainer" containerID="d188f3b96d80ea0803499f9daad90b489848001b0d14c7e58153e27bb2089fef"
Jul 16 22:23:18.249463 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:18.249426    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:23:18.256001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:18.255973    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-8htfn"]
Jul 16 22:23:19.155513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:19.155479    2581 scope.go:117] "RemoveContainer" containerID="a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb"
Jul 16 22:23:19.160901 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:19.160877    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" path="/var/lib/kubelet/pods/5152ba4c-3a52-4e1b-b804-3450be12535b/volumes"
Jul 16 22:23:20.214683 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:20.214598    2581 generic.go:356] "Generic (PLEG): container finished" podID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerID="fc1c5f65055e283a86b23aea65678962164f03407aa18ca0aa06da0d8810f5ea" exitCode=6
Jul 16 22:23:20.214683 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:20.214665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" event={"ID":"e3c13108-8ea1-4fff-855a-3cdf27187cf9","Type":"ContainerDied","Data":"fc1c5f65055e283a86b23aea65678962164f03407aa18ca0aa06da0d8810f5ea"}
Jul 16 22:23:20.215140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:20.214735    2581 scope.go:117] "RemoveContainer" containerID="a9f185f92bd62ed398a4b27a04972a1b4927d2e535b68717f7138f9e7365b9eb"
Jul 16 22:23:20.215140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:20.215003    2581 scope.go:117] "RemoveContainer" containerID="fc1c5f65055e283a86b23aea65678962164f03407aa18ca0aa06da0d8810f5ea"
Jul 16 22:23:20.215215 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:20.215178    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn_odh-ai-gateway-infra(e3c13108-8ea1-4fff-855a-3cdf27187cf9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9"
Jul 16 22:23:21.257303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:21.257264    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:21.395106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:21.395080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:21.508325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:21.508239    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca\") pod \"e3c13108-8ea1-4fff-855a-3cdf27187cf9\" (UID: \"e3c13108-8ea1-4fff-855a-3cdf27187cf9\") "
Jul 16 22:23:21.508519 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:21.508498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e3c13108-8ea1-4fff-855a-3cdf27187cf9" (UID: "e3c13108-8ea1-4fff-855a-3cdf27187cf9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:21.609656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:21.609622    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e3c13108-8ea1-4fff-855a-3cdf27187cf9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:23:22.227092 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:22.227061    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"
Jul 16 22:23:22.227092 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:22.227072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn" event={"ID":"e3c13108-8ea1-4fff-855a-3cdf27187cf9","Type":"ContainerDied","Data":"64119384f75b821d317ebf81117077d6514a0bd8fd6f3e5ed8fa3f639f12e652"}
Jul 16 22:23:22.227399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:22.227129    2581 scope.go:117] "RemoveContainer" containerID="fc1c5f65055e283a86b23aea65678962164f03407aa18ca0aa06da0d8810f5ea"
Jul 16 22:23:22.250315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:22.250288    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:22.257315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:22.257291    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-d75jn"]
Jul 16 22:23:23.160559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:23.160523    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" path="/var/lib/kubelet/pods/e3c13108-8ea1-4fff-855a-3cdf27187cf9/volumes"
Jul 16 22:23:34.020386 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020343    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020676    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020697    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020703    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020708    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020713    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020725    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020730    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="827b6be9-dd63-4a6d-872b-e6570154b5a9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020741    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020746    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020796    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.020801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020803    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.021223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020810    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.021223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020816    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:34.021223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020821    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.021223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.020827    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:34.023417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.023401    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:34.027080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.027060    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:23:34.027287 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.027260    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:23:34.080266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.080217    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:34.125038 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.124993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf\" (UID: \"423c9c35-1c9f-4703-b8ab-eeef38ecccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:34.226382 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.226333    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf\" (UID: \"423c9c35-1c9f-4703-b8ab-eeef38ecccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:34.226946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.226927    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf\" (UID: \"423c9c35-1c9f-4703-b8ab-eeef38ecccc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:34.334187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.334088    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:34.473782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:34.473757    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:34.476478 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:23:34.476451    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod423c9c35_1c9f_4703_b8ab_eeef38ecccc4.slice/crio-53be95d010ab6bc2069050844cfbfe0b2e1caeeed9d139c1e4f83bdafd2406f3 WatchSource:0}: Error finding container 53be95d010ab6bc2069050844cfbfe0b2e1caeeed9d139c1e4f83bdafd2406f3: Status 404 returned error can't find the container with id 53be95d010ab6bc2069050844cfbfe0b2e1caeeed9d139c1e4f83bdafd2406f3
Jul 16 22:23:35.285865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:35.285827    2581 generic.go:356] "Generic (PLEG): container finished" podID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerID="a390769bc5090846026f7318741bb6f336df5a0ee72e56eba3848788fe1e47f2" exitCode=7
Jul 16 22:23:35.286368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:35.285923    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" event={"ID":"423c9c35-1c9f-4703-b8ab-eeef38ecccc4","Type":"ContainerDied","Data":"a390769bc5090846026f7318741bb6f336df5a0ee72e56eba3848788fe1e47f2"}
Jul 16 22:23:35.286368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:35.285977    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" event={"ID":"423c9c35-1c9f-4703-b8ab-eeef38ecccc4","Type":"ContainerStarted","Data":"53be95d010ab6bc2069050844cfbfe0b2e1caeeed9d139c1e4f83bdafd2406f3"}
Jul 16 22:23:35.286368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:35.286098    2581 scope.go:117] "RemoveContainer" containerID="a390769bc5090846026f7318741bb6f336df5a0ee72e56eba3848788fe1e47f2"
Jul 16 22:23:36.292385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:36.292288    2581 generic.go:356] "Generic (PLEG): container finished" podID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerID="fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac" exitCode=7
Jul 16 22:23:36.292385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:36.292372    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" event={"ID":"423c9c35-1c9f-4703-b8ab-eeef38ecccc4","Type":"ContainerDied","Data":"fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac"}
Jul 16 22:23:36.292842 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:36.292442    2581 scope.go:117] "RemoveContainer" containerID="a390769bc5090846026f7318741bb6f336df5a0ee72e56eba3848788fe1e47f2"
Jul 16 22:23:36.292842 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:36.292687    2581 scope.go:117] "RemoveContainer" containerID="fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac"
Jul 16 22:23:36.292947 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:36.292872    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf_odh-ai-gateway-infra(423c9c35-1c9f-4703-b8ab-eeef38ecccc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4"
Jul 16 22:23:37.298972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:37.298941    2581 scope.go:117] "RemoveContainer" containerID="fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac"
Jul 16 22:23:37.299383 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:37.299119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf_odh-ai-gateway-infra(423c9c35-1c9f-4703-b8ab-eeef38ecccc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4"
Jul 16 22:23:50.155584 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:50.155555    2581 scope.go:117] "RemoveContainer" containerID="fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac"
Jul 16 22:23:51.367897 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:51.367860    2581 generic.go:356] "Generic (PLEG): container finished" podID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerID="f36613792d2100bd35ac6b259ef0dbf9cfacbd886b00774ec5aaf2ef3c7d3b79" exitCode=7
Jul 16 22:23:51.368418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:51.367931    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" event={"ID":"423c9c35-1c9f-4703-b8ab-eeef38ecccc4","Type":"ContainerDied","Data":"f36613792d2100bd35ac6b259ef0dbf9cfacbd886b00774ec5aaf2ef3c7d3b79"}
Jul 16 22:23:51.368418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:51.367991    2581 scope.go:117] "RemoveContainer" containerID="fb2993ec813db51d73a5cfac8fddc3b0363a07e612096fc25e7f62ef4ca24dac"
Jul 16 22:23:51.368418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:51.368309    2581 scope.go:117] "RemoveContainer" containerID="f36613792d2100bd35ac6b259ef0dbf9cfacbd886b00774ec5aaf2ef3c7d3b79"
Jul 16 22:23:51.368600 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:51.368575    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf_odh-ai-gateway-infra(423c9c35-1c9f-4703-b8ab-eeef38ecccc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4"
Jul 16 22:23:52.394119 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:52.394086    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:52.525307 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:52.525280    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:52.587828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:52.587801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca\") pod \"423c9c35-1c9f-4703-b8ab-eeef38ecccc4\" (UID: \"423c9c35-1c9f-4703-b8ab-eeef38ecccc4\") "
Jul 16 22:23:52.588193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:52.588174    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "423c9c35-1c9f-4703-b8ab-eeef38ecccc4" (UID: "423c9c35-1c9f-4703-b8ab-eeef38ecccc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:52.689204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:52.689137    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/423c9c35-1c9f-4703-b8ab-eeef38ecccc4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:23:53.379403 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:53.379363    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf" event={"ID":"423c9c35-1c9f-4703-b8ab-eeef38ecccc4","Type":"ContainerDied","Data":"53be95d010ab6bc2069050844cfbfe0b2e1caeeed9d139c1e4f83bdafd2406f3"}
Jul 16 22:23:53.379403 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:53.379394    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"
Jul 16 22:23:53.379575 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:53.379433    2581 scope.go:117] "RemoveContainer" containerID="f36613792d2100bd35ac6b259ef0dbf9cfacbd886b00774ec5aaf2ef3c7d3b79"
Jul 16 22:23:53.395845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:53.395821    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:53.400012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:53.399990    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-f22qf"]
Jul 16 22:23:54.835556 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835522    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835887    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835898    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3c13108-8ea1-4fff-855a-3cdf27187cf9" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835912    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835918    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835927    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835932    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5152ba4c-3a52-4e1b-b804-3450be12535b" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835938    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.835943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835943    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.836192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835949    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.836192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.835955    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.836192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.836002    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.836192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.836010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.836192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.836018    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" containerName="revoke-keys"
Jul 16 22:23:54.840576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.840558    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:23:54.843015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.842999    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:23:54.843117 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.843094    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:23:54.846380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.846322    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:23:54.908573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:54.908542    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr\" (UID: \"7aed0968-f265-4187-84e1-f6690ad71c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:23:55.009308 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.009276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr\" (UID: \"7aed0968-f265-4187-84e1-f6690ad71c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:23:55.009891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.009871    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr\" (UID: \"7aed0968-f265-4187-84e1-f6690ad71c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:23:55.152555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.152522    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:23:55.160368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.160337    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="423c9c35-1c9f-4703-b8ab-eeef38ecccc4" path="/var/lib/kubelet/pods/423c9c35-1c9f-4703-b8ab-eeef38ecccc4/volumes"
Jul 16 22:23:55.289135 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.289106    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:23:55.291399 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:23:55.291361    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7aed0968_f265_4187_84e1_f6690ad71c11.slice/crio-4d1cdf504c7a58988df2288fa11c3f895be3a874f15025d19caceb8d9236a916 WatchSource:0}: Error finding container 4d1cdf504c7a58988df2288fa11c3f895be3a874f15025d19caceb8d9236a916: Status 404 returned error can't find the container with id 4d1cdf504c7a58988df2288fa11c3f895be3a874f15025d19caceb8d9236a916
Jul 16 22:23:55.390462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:55.390436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerStarted","Data":"4d1cdf504c7a58988df2288fa11c3f895be3a874f15025d19caceb8d9236a916"}
Jul 16 22:23:56.390655 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.390623    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:23:56.394352 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.394327    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:23:56.396451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.396422    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aed0968-f265-4187-84e1-f6690ad71c11" containerID="4e663e94562a069d0b4815bb21d58a5877ce17611ae6a16a52b3c5c5bcaa75ef" exitCode=6
Jul 16 22:23:56.396555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.396509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerDied","Data":"4e663e94562a069d0b4815bb21d58a5877ce17611ae6a16a52b3c5c5bcaa75ef"}
Jul 16 22:23:56.396663 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.396650    2581 scope.go:117] "RemoveContainer" containerID="4e663e94562a069d0b4815bb21d58a5877ce17611ae6a16a52b3c5c5bcaa75ef"
Jul 16 22:23:56.400922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.400854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:23:56.421017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.420985    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c\" (UID: \"977d2fb4-f49f-4452-8467-472a51ccbe08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:23:56.522370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.522334    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c\" (UID: \"977d2fb4-f49f-4452-8467-472a51ccbe08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:23:56.522952 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.522932    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c\" (UID: \"977d2fb4-f49f-4452-8467-472a51ccbe08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:23:56.706213 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.706134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:23:56.834841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:56.834813    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:23:56.837014 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:23:56.836987    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod977d2fb4_f49f_4452_8467_472a51ccbe08.slice/crio-4cd432e5b90797dec3efebc174fac642869fe58327f8df7a7ce863b631d599c2 WatchSource:0}: Error finding container 4cd432e5b90797dec3efebc174fac642869fe58327f8df7a7ce863b631d599c2: Status 404 returned error can't find the container with id 4cd432e5b90797dec3efebc174fac642869fe58327f8df7a7ce863b631d599c2
Jul 16 22:23:57.403212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.403173    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aed0968-f265-4187-84e1-f6690ad71c11" containerID="16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455" exitCode=6
Jul 16 22:23:57.403678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.403263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerDied","Data":"16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455"}
Jul 16 22:23:57.403678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.403320    2581 scope.go:117] "RemoveContainer" containerID="4e663e94562a069d0b4815bb21d58a5877ce17611ae6a16a52b3c5c5bcaa75ef"
Jul 16 22:23:57.403678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.403505    2581 scope.go:117] "RemoveContainer" containerID="16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455"
Jul 16 22:23:57.403845 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:57.403752    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr_odh-ai-gateway-infra(7aed0968-f265-4187-84e1-f6690ad71c11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" podUID="7aed0968-f265-4187-84e1-f6690ad71c11"
Jul 16 22:23:57.404843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.404812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerStarted","Data":"ce99bd79343a21c773449f96bee2119820131088dcdaae0ac48ef873cd2cdfeb"}
Jul 16 22:23:57.404913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.404857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerStarted","Data":"4cd432e5b90797dec3efebc174fac642869fe58327f8df7a7ce863b631d599c2"}
Jul 16 22:23:57.437843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:57.437801    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" podStartSLOduration=1.437788882 podStartE2EDuration="1.437788882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:57.435430124 +0000 UTC m=+2224.764870027" watchObservedRunningTime="2026-07-16 22:23:57.437788882 +0000 UTC m=+2224.767228784"
Jul 16 22:23:58.410438 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:58.410402    2581 generic.go:356] "Generic (PLEG): container finished" podID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerID="ce99bd79343a21c773449f96bee2119820131088dcdaae0ac48ef873cd2cdfeb" exitCode=6
Jul 16 22:23:58.410868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:58.410490    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerDied","Data":"ce99bd79343a21c773449f96bee2119820131088dcdaae0ac48ef873cd2cdfeb"}
Jul 16 22:23:58.410868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:58.410624    2581 scope.go:117] "RemoveContainer" containerID="ce99bd79343a21c773449f96bee2119820131088dcdaae0ac48ef873cd2cdfeb"
Jul 16 22:23:58.412283 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:58.412268    2581 scope.go:117] "RemoveContainer" containerID="16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455"
Jul 16 22:23:58.412438 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:58.412421    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr_odh-ai-gateway-infra(7aed0968-f265-4187-84e1-f6690ad71c11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" podUID="7aed0968-f265-4187-84e1-f6690ad71c11"
Jul 16 22:23:59.417851 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:59.417813    2581 generic.go:356] "Generic (PLEG): container finished" podID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerID="47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af" exitCode=6
Jul 16 22:23:59.418366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:59.417860    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerDied","Data":"47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af"}
Jul 16 22:23:59.418366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:59.417908    2581 scope.go:117] "RemoveContainer" containerID="ce99bd79343a21c773449f96bee2119820131088dcdaae0ac48ef873cd2cdfeb"
Jul 16 22:23:59.418366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:23:59.418136    2581 scope.go:117] "RemoveContainer" containerID="47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af"
Jul 16 22:23:59.418528 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:23:59.418407    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c_odh-ai-gateway-infra(977d2fb4-f49f-4452-8467-472a51ccbe08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08"
Jul 16 22:24:00.424205 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:00.424179    2581 scope.go:117] "RemoveContainer" containerID="47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af"
Jul 16 22:24:00.424593 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:00.424380    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c_odh-ai-gateway-infra(977d2fb4-f49f-4452-8467-472a51ccbe08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08"
Jul 16 22:24:11.155153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:11.155123    2581 scope.go:117] "RemoveContainer" containerID="16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455"
Jul 16 22:24:11.473586 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:11.473509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerStarted","Data":"0034211d3b7f2a055504dea5304b31537ff164817679a4e119ac19b4018dafa4"}
Jul 16 22:24:11.494977 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:11.494929    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" podStartSLOduration=17.494913259 podStartE2EDuration="17.494913259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:11.492607168 +0000 UTC m=+2238.822047070" watchObservedRunningTime="2026-07-16 22:24:11.494913259 +0000 UTC m=+2238.824353156"
Jul 16 22:24:12.480096 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:12.480062    2581 generic.go:356] "Generic (PLEG): container finished" podID="7aed0968-f265-4187-84e1-f6690ad71c11" containerID="0034211d3b7f2a055504dea5304b31537ff164817679a4e119ac19b4018dafa4" exitCode=6
Jul 16 22:24:12.480597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:12.480115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerDied","Data":"0034211d3b7f2a055504dea5304b31537ff164817679a4e119ac19b4018dafa4"}
Jul 16 22:24:12.480597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:12.480164    2581 scope.go:117] "RemoveContainer" containerID="16fd4fa715f5ab3c87e84a58ec2ee41d1ffb83e9546e6bdfa1d5833563bf5455"
Jul 16 22:24:12.480597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:12.480444    2581 scope.go:117] "RemoveContainer" containerID="0034211d3b7f2a055504dea5304b31537ff164817679a4e119ac19b4018dafa4"
Jul 16 22:24:12.480707 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:12.480650    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr_odh-ai-gateway-infra(7aed0968-f265-4187-84e1-f6690ad71c11)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" podUID="7aed0968-f265-4187-84e1-f6690ad71c11"
Jul 16 22:24:12.519343 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:12.519310    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:24:13.622241 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:13.622207    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:24:13.766125 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:13.766014    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca\") pod \"7aed0968-f265-4187-84e1-f6690ad71c11\" (UID: \"7aed0968-f265-4187-84e1-f6690ad71c11\") "
Jul 16 22:24:13.766423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:13.766400    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7aed0968-f265-4187-84e1-f6690ad71c11" (UID: "7aed0968-f265-4187-84e1-f6690ad71c11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:13.866923 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:13.866892    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7aed0968-f265-4187-84e1-f6690ad71c11-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:24:14.155319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.155284    2581 scope.go:117] "RemoveContainer" containerID="47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af"
Jul 16 22:24:14.493388 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.493350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerStarted","Data":"6c98aae14a8cb7962228881521394fbcb04a905bc6bef8c93b1153bd670bc5f9"}
Jul 16 22:24:14.494626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.494598    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"
Jul 16 22:24:14.494626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.494613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr" event={"ID":"7aed0968-f265-4187-84e1-f6690ad71c11","Type":"ContainerDied","Data":"4d1cdf504c7a58988df2288fa11c3f895be3a874f15025d19caceb8d9236a916"}
Jul 16 22:24:14.494793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.494670    2581 scope.go:117] "RemoveContainer" containerID="0034211d3b7f2a055504dea5304b31537ff164817679a4e119ac19b4018dafa4"
Jul 16 22:24:14.522324 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.522297    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:24:14.526209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:14.526182    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gnvjr"]
Jul 16 22:24:15.160304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.160269    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" path="/var/lib/kubelet/pods/7aed0968-f265-4187-84e1-f6690ad71c11/volumes"
Jul 16 22:24:15.501346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.501257    2581 generic.go:356] "Generic (PLEG): container finished" podID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerID="6c98aae14a8cb7962228881521394fbcb04a905bc6bef8c93b1153bd670bc5f9" exitCode=6
Jul 16 22:24:15.501346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.501300    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerDied","Data":"6c98aae14a8cb7962228881521394fbcb04a905bc6bef8c93b1153bd670bc5f9"}
Jul 16 22:24:15.501346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.501347    2581 scope.go:117] "RemoveContainer" containerID="47c1d2a63add234a71523ea0a557d9c7d081d1d5eeb5a14f1232ccbc2936e3af"
Jul 16 22:24:15.501626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.501598    2581 scope.go:117] "RemoveContainer" containerID="6c98aae14a8cb7962228881521394fbcb04a905bc6bef8c93b1153bd670bc5f9"
Jul 16 22:24:15.501836 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:15.501816    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c_odh-ai-gateway-infra(977d2fb4-f49f-4452-8467-472a51ccbe08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08"
Jul 16 22:24:15.528466 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:15.528436    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:24:16.637270 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:16.637246    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:24:16.792141 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:16.792038    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca\") pod \"977d2fb4-f49f-4452-8467-472a51ccbe08\" (UID: \"977d2fb4-f49f-4452-8467-472a51ccbe08\") "
Jul 16 22:24:16.792428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:16.792406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "977d2fb4-f49f-4452-8467-472a51ccbe08" (UID: "977d2fb4-f49f-4452-8467-472a51ccbe08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:16.893046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:16.893004    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/977d2fb4-f49f-4452-8467-472a51ccbe08-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:24:17.513089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:17.513006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"
Jul 16 22:24:17.513274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:17.513006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c" event={"ID":"977d2fb4-f49f-4452-8467-472a51ccbe08","Type":"ContainerDied","Data":"4cd432e5b90797dec3efebc174fac642869fe58327f8df7a7ce863b631d599c2"}
Jul 16 22:24:17.513274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:17.513142    2581 scope.go:117] "RemoveContainer" containerID="6c98aae14a8cb7962228881521394fbcb04a905bc6bef8c93b1153bd670bc5f9"
Jul 16 22:24:17.558303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:17.558272    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:24:17.566565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:17.566543    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-p7g7c"]
Jul 16 22:24:19.160311 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:19.160278    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" path="/var/lib/kubelet/pods/977d2fb4-f49f-4452-8467-472a51ccbe08/volumes"
Jul 16 22:24:24.059809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.059775    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060162    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060175    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060184    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060202    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060208    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060214    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060219    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060307    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060315    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060321    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.060327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.060327    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:24.064889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.064872    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:24.067394 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.067366    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:24.067473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.067432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:24:24.073308 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.073286    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:24.158765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.158729    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2\" (UID: \"e7afd4a2-e8f3-4a51-a124-94a30694098e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:24.260305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.260275    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2\" (UID: \"e7afd4a2-e8f3-4a51-a124-94a30694098e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:24.260852 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.260831    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2\" (UID: \"e7afd4a2-e8f3-4a51-a124-94a30694098e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:24.376735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.376712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:24.509311 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.509222    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:24.511486 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:24:24.511449    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7afd4a2_e8f3_4a51_a124_94a30694098e.slice/crio-f96a9b95ea370bce750a5d613a15b9ccc066127c7c034558e0b517e30cce41f6 WatchSource:0}: Error finding container f96a9b95ea370bce750a5d613a15b9ccc066127c7c034558e0b517e30cce41f6: Status 404 returned error can't find the container with id f96a9b95ea370bce750a5d613a15b9ccc066127c7c034558e0b517e30cce41f6
Jul 16 22:24:24.546606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:24.546576    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerStarted","Data":"f96a9b95ea370bce750a5d613a15b9ccc066127c7c034558e0b517e30cce41f6"}
Jul 16 22:24:25.552408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:25.552318    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerID="67503d5bf931dc81cad879711aa50a5517b59a233c4b35bb5144f602c16b8d53" exitCode=7
Jul 16 22:24:25.552408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:25.552356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerDied","Data":"67503d5bf931dc81cad879711aa50a5517b59a233c4b35bb5144f602c16b8d53"}
Jul 16 22:24:25.552866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:25.552544    2581 scope.go:117] "RemoveContainer" containerID="67503d5bf931dc81cad879711aa50a5517b59a233c4b35bb5144f602c16b8d53"
Jul 16 22:24:26.558558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:26.558462    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerID="a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c" exitCode=7
Jul 16 22:24:26.558558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:26.558506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerDied","Data":"a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c"}
Jul 16 22:24:26.558558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:26.558556    2581 scope.go:117] "RemoveContainer" containerID="67503d5bf931dc81cad879711aa50a5517b59a233c4b35bb5144f602c16b8d53"
Jul 16 22:24:26.559086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:26.558739    2581 scope.go:117] "RemoveContainer" containerID="a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c"
Jul 16 22:24:26.559086 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:26.558948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2_odh-ai-gateway-infra(e7afd4a2-e8f3-4a51-a124-94a30694098e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e"
Jul 16 22:24:27.564458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:27.564426    2581 scope.go:117] "RemoveContainer" containerID="a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c"
Jul 16 22:24:27.564905 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:27.564662    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2_odh-ai-gateway-infra(e7afd4a2-e8f3-4a51-a124-94a30694098e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e"
Jul 16 22:24:41.154843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:41.154812    2581 scope.go:117] "RemoveContainer" containerID="a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c"
Jul 16 22:24:41.627726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:41.627695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerStarted","Data":"40856f323ca1c0c4cae42132e2f4ecdcebe240b569fda1fc52fa5d362c2fb189"}
Jul 16 22:24:41.643943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:41.643879    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" podStartSLOduration=17.643862086 podStartE2EDuration="17.643862086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:41.643525433 +0000 UTC m=+2268.972965334" watchObservedRunningTime="2026-07-16 22:24:41.643862086 +0000 UTC m=+2268.973301988"
Jul 16 22:24:42.633809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:42.633776    2581 generic.go:356] "Generic (PLEG): container finished" podID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerID="40856f323ca1c0c4cae42132e2f4ecdcebe240b569fda1fc52fa5d362c2fb189" exitCode=7
Jul 16 22:24:42.634319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:42.633851    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerDied","Data":"40856f323ca1c0c4cae42132e2f4ecdcebe240b569fda1fc52fa5d362c2fb189"}
Jul 16 22:24:42.634319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:42.633913    2581 scope.go:117] "RemoveContainer" containerID="a0d0a8cb66ab8bdff4deaeade5198248e983e471e0b0a5802d59b44212b8261c"
Jul 16 22:24:42.634319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:42.634173    2581 scope.go:117] "RemoveContainer" containerID="40856f323ca1c0c4cae42132e2f4ecdcebe240b569fda1fc52fa5d362c2fb189"
Jul 16 22:24:42.634533 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:42.634395    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2_odh-ai-gateway-infra(e7afd4a2-e8f3-4a51-a124-94a30694098e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e"
Jul 16 22:24:42.661224 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:42.661200    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:43.768712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:43.768690    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:43.818968 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:43.818941    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca\") pod \"e7afd4a2-e8f3-4a51-a124-94a30694098e\" (UID: \"e7afd4a2-e8f3-4a51-a124-94a30694098e\") "
Jul 16 22:24:43.819327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:43.819303    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7afd4a2-e8f3-4a51-a124-94a30694098e" (UID: "e7afd4a2-e8f3-4a51-a124-94a30694098e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:43.920049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:43.919972    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7afd4a2-e8f3-4a51-a124-94a30694098e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:24:44.644999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.644968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"
Jul 16 22:24:44.645204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.644972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2" event={"ID":"e7afd4a2-e8f3-4a51-a124-94a30694098e","Type":"ContainerDied","Data":"f96a9b95ea370bce750a5d613a15b9ccc066127c7c034558e0b517e30cce41f6"}
Jul 16 22:24:44.645204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.645108    2581 scope.go:117] "RemoveContainer" containerID="40856f323ca1c0c4cae42132e2f4ecdcebe240b569fda1fc52fa5d362c2fb189"
Jul 16 22:24:44.668620 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.668597    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:44.676397 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.676371    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8tkf2"]
Jul 16 22:24:44.878747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.878714    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879013    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879047    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879083    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879101    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879107    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:44.879109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879112    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7aed0968-f265-4187-84e1-f6690ad71c11" containerName="revoke-keys"
Jul 16 22:24:44.879383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879118    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879125    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879167    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879175    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:44.879383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.879184    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="977d2fb4-f49f-4452-8467-472a51ccbe08" containerName="revoke-keys"
Jul 16 22:24:44.883377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.883361    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:24:44.885682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.885664    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:44.885788 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.885663    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:24:44.895219 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.895155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:24:44.928431 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:44.928401    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2\" (UID: \"032f45b2-772e-4067-837d-ac7aae291cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:24:45.029293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.029259    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2\" (UID: \"032f45b2-772e-4067-837d-ac7aae291cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:24:45.029867 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.029842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2\" (UID: \"032f45b2-772e-4067-837d-ac7aae291cc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:24:45.162418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.162339    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" path="/var/lib/kubelet/pods/e7afd4a2-e8f3-4a51-a124-94a30694098e/volumes"
Jul 16 22:24:45.200353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.200323    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:24:45.328912 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.328886    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:24:45.331266 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:24:45.331236    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod032f45b2_772e_4067_837d_ac7aae291cc2.slice/crio-64a6b5c9b5adbd8957ff688f2e5318994de019aa9a80997394fc6a58382c2245 WatchSource:0}: Error finding container 64a6b5c9b5adbd8957ff688f2e5318994de019aa9a80997394fc6a58382c2245: Status 404 returned error can't find the container with id 64a6b5c9b5adbd8957ff688f2e5318994de019aa9a80997394fc6a58382c2245
Jul 16 22:24:45.650890 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.650848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerStarted","Data":"4a58a6e86a1e98bfb66727eae1829788d2efafc4fb3522732214a2b66bed2909"}
Jul 16 22:24:45.651067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.650914    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerStarted","Data":"64a6b5c9b5adbd8957ff688f2e5318994de019aa9a80997394fc6a58382c2245"}
Jul 16 22:24:45.665828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:45.665780    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" podStartSLOduration=1.665765122 podStartE2EDuration="1.665765122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:45.664200289 +0000 UTC m=+2272.993640190" watchObservedRunningTime="2026-07-16 22:24:45.665765122 +0000 UTC m=+2272.995205024"
Jul 16 22:24:46.658046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:46.658008    2581 generic.go:356] "Generic (PLEG): container finished" podID="032f45b2-772e-4067-837d-ac7aae291cc2" containerID="4a58a6e86a1e98bfb66727eae1829788d2efafc4fb3522732214a2b66bed2909" exitCode=6
Jul 16 22:24:46.658450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:46.658095    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerDied","Data":"4a58a6e86a1e98bfb66727eae1829788d2efafc4fb3522732214a2b66bed2909"}
Jul 16 22:24:46.658450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:46.658292    2581 scope.go:117] "RemoveContainer" containerID="4a58a6e86a1e98bfb66727eae1829788d2efafc4fb3522732214a2b66bed2909"
Jul 16 22:24:47.663748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:47.663714    2581 generic.go:356] "Generic (PLEG): container finished" podID="032f45b2-772e-4067-837d-ac7aae291cc2" containerID="df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a" exitCode=6
Jul 16 22:24:47.664149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:47.663783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerDied","Data":"df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a"}
Jul 16 22:24:47.664149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:47.663842    2581 scope.go:117] "RemoveContainer" containerID="4a58a6e86a1e98bfb66727eae1829788d2efafc4fb3522732214a2b66bed2909"
Jul 16 22:24:47.664149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:47.663993    2581 scope.go:117] "RemoveContainer" containerID="df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a"
Jul 16 22:24:47.664294 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:47.664201    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2_odh-ai-gateway-infra(032f45b2-772e-4067-837d-ac7aae291cc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" podUID="032f45b2-772e-4067-837d-ac7aae291cc2"
Jul 16 22:24:48.670155 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:48.670126    2581 scope.go:117] "RemoveContainer" containerID="df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a"
Jul 16 22:24:48.670543 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:48.670346    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2_odh-ai-gateway-infra(032f45b2-772e-4067-837d-ac7aae291cc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" podUID="032f45b2-772e-4067-837d-ac7aae291cc2"
Jul 16 22:24:56.438684 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.438643    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:24:56.439121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.439040    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7afd4a2-e8f3-4a51-a124-94a30694098e" containerName="revoke-keys"
Jul 16 22:24:56.443558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.443541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:24:56.452655 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.452624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:24:56.531298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.531254    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg\" (UID: \"a3eadb2d-0415-48c9-acc6-8604c3d61d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:24:56.632300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.632260    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg\" (UID: \"a3eadb2d-0415-48c9-acc6-8604c3d61d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:24:56.632841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.632821    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg\" (UID: \"a3eadb2d-0415-48c9-acc6-8604c3d61d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:24:56.754632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.754541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:24:56.881378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:56.881355    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:24:56.883343 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:24:56.883301    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3eadb2d_0415_48c9_acc6_8604c3d61d9a.slice/crio-39b487e70beb4a4775c7eba01f62b678b02af758d1bb422b05ce718a30183fbd WatchSource:0}: Error finding container 39b487e70beb4a4775c7eba01f62b678b02af758d1bb422b05ce718a30183fbd: Status 404 returned error can't find the container with id 39b487e70beb4a4775c7eba01f62b678b02af758d1bb422b05ce718a30183fbd
Jul 16 22:24:57.717662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:57.717560    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerID="422aaf40ac92dcc76ff0833ae3982f9cb26c2577ed4980f88a9b2f5c397ef1bd" exitCode=6
Jul 16 22:24:57.717662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:57.717613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" event={"ID":"a3eadb2d-0415-48c9-acc6-8604c3d61d9a","Type":"ContainerDied","Data":"422aaf40ac92dcc76ff0833ae3982f9cb26c2577ed4980f88a9b2f5c397ef1bd"}
Jul 16 22:24:57.718163 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:57.717670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" event={"ID":"a3eadb2d-0415-48c9-acc6-8604c3d61d9a","Type":"ContainerStarted","Data":"39b487e70beb4a4775c7eba01f62b678b02af758d1bb422b05ce718a30183fbd"}
Jul 16 22:24:57.718163 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:57.717925    2581 scope.go:117] "RemoveContainer" containerID="422aaf40ac92dcc76ff0833ae3982f9cb26c2577ed4980f88a9b2f5c397ef1bd"
Jul 16 22:24:58.723686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:58.723592    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerID="fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68" exitCode=6
Jul 16 22:24:58.723686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:58.723670    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" event={"ID":"a3eadb2d-0415-48c9-acc6-8604c3d61d9a","Type":"ContainerDied","Data":"fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68"}
Jul 16 22:24:58.724150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:58.723725    2581 scope.go:117] "RemoveContainer" containerID="422aaf40ac92dcc76ff0833ae3982f9cb26c2577ed4980f88a9b2f5c397ef1bd"
Jul 16 22:24:58.724150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:58.723913    2581 scope.go:117] "RemoveContainer" containerID="fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68"
Jul 16 22:24:58.724150 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:58.724125    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg_odh-ai-gateway-infra(a3eadb2d-0415-48c9-acc6-8604c3d61d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a"
Jul 16 22:24:59.729977 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:24:59.729949    2581 scope.go:117] "RemoveContainer" containerID="fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68"
Jul 16 22:24:59.730440 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:24:59.730202    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg_odh-ai-gateway-infra(a3eadb2d-0415-48c9-acc6-8604c3d61d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a"
Jul 16 22:25:03.157551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:03.157527    2581 scope.go:117] "RemoveContainer" containerID="df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a"
Jul 16 22:25:03.749987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:03.749955    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerStarted","Data":"7d6f50bbb4e6d58159c9c659295ca9cc41ddea810796e5da790967fe77eec692"}
Jul 16 22:25:04.756505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:04.756474    2581 generic.go:356] "Generic (PLEG): container finished" podID="032f45b2-772e-4067-837d-ac7aae291cc2" containerID="7d6f50bbb4e6d58159c9c659295ca9cc41ddea810796e5da790967fe77eec692" exitCode=6
Jul 16 22:25:04.756919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:04.756541    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerDied","Data":"7d6f50bbb4e6d58159c9c659295ca9cc41ddea810796e5da790967fe77eec692"}
Jul 16 22:25:04.756919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:04.756590    2581 scope.go:117] "RemoveContainer" containerID="df473cf961bc895f5529e749da882195cbe24629950c3c403040d07972ed0b2a"
Jul 16 22:25:04.756919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:04.756874    2581 scope.go:117] "RemoveContainer" containerID="7d6f50bbb4e6d58159c9c659295ca9cc41ddea810796e5da790967fe77eec692"
Jul 16 22:25:04.757080 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:04.757062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2_odh-ai-gateway-infra(032f45b2-772e-4067-837d-ac7aae291cc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" podUID="032f45b2-772e-4067-837d-ac7aae291cc2"
Jul 16 22:25:04.788202 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:04.788174    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:25:05.893359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:05.893337    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:25:06.011394 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.011364    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca\") pod \"032f45b2-772e-4067-837d-ac7aae291cc2\" (UID: \"032f45b2-772e-4067-837d-ac7aae291cc2\") "
Jul 16 22:25:06.011695 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.011673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "032f45b2-772e-4067-837d-ac7aae291cc2" (UID: "032f45b2-772e-4067-837d-ac7aae291cc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:06.112056 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.112027    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/032f45b2-772e-4067-837d-ac7aae291cc2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:25:06.768488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.768457    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"
Jul 16 22:25:06.768488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.768468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2" event={"ID":"032f45b2-772e-4067-837d-ac7aae291cc2","Type":"ContainerDied","Data":"64a6b5c9b5adbd8957ff688f2e5318994de019aa9a80997394fc6a58382c2245"}
Jul 16 22:25:06.768755 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.768545    2581 scope.go:117] "RemoveContainer" containerID="7d6f50bbb4e6d58159c9c659295ca9cc41ddea810796e5da790967fe77eec692"
Jul 16 22:25:06.791528 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.791503    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:25:06.797064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:06.797036    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-z4rh2"]
Jul 16 22:25:07.159893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:07.159861    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" path="/var/lib/kubelet/pods/032f45b2-772e-4067-837d-ac7aae291cc2/volumes"
Jul 16 22:25:11.155465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:11.155434    2581 scope.go:117] "RemoveContainer" containerID="fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68"
Jul 16 22:25:11.792469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:11.792438    2581 generic.go:356] "Generic (PLEG): container finished" podID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerID="cfacecee5fe907da5d53f986c30c0174403e13c31d5fa50f5d2e399df0b69c96" exitCode=6
Jul 16 22:25:11.792608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:11.792510    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" event={"ID":"a3eadb2d-0415-48c9-acc6-8604c3d61d9a","Type":"ContainerDied","Data":"cfacecee5fe907da5d53f986c30c0174403e13c31d5fa50f5d2e399df0b69c96"}
Jul 16 22:25:11.792608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:11.792570    2581 scope.go:117] "RemoveContainer" containerID="fe89c7f81b1ed9ce10a0b7276aeb706c3c363a4531eecb51e7ec2d0c49311a68"
Jul 16 22:25:11.792925 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:11.792904    2581 scope.go:117] "RemoveContainer" containerID="cfacecee5fe907da5d53f986c30c0174403e13c31d5fa50f5d2e399df0b69c96"
Jul 16 22:25:11.793138 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:11.793119    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg_odh-ai-gateway-infra(a3eadb2d-0415-48c9-acc6-8604c3d61d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a"
Jul 16 22:25:12.818246 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:12.818181    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:25:12.949222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:12.949199    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:25:13.072442 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.072361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca\") pod \"a3eadb2d-0415-48c9-acc6-8604c3d61d9a\" (UID: \"a3eadb2d-0415-48c9-acc6-8604c3d61d9a\") "
Jul 16 22:25:13.072707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.072684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a3eadb2d-0415-48c9-acc6-8604c3d61d9a" (UID: "a3eadb2d-0415-48c9-acc6-8604c3d61d9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:13.172994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.172965    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3eadb2d-0415-48c9-acc6-8604c3d61d9a-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:25:13.803888 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.803858    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg" event={"ID":"a3eadb2d-0415-48c9-acc6-8604c3d61d9a","Type":"ContainerDied","Data":"39b487e70beb4a4775c7eba01f62b678b02af758d1bb422b05ce718a30183fbd"}
Jul 16 22:25:13.803888 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.803877    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"
Jul 16 22:25:13.804147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.803920    2581 scope.go:117] "RemoveContainer" containerID="cfacecee5fe907da5d53f986c30c0174403e13c31d5fa50f5d2e399df0b69c96"
Jul 16 22:25:13.852510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.852482    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:25:13.858895 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:13.858868    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-hdsvg"]
Jul 16 22:25:15.160014 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:15.159970    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" path="/var/lib/kubelet/pods/a3eadb2d-0415-48c9-acc6-8604c3d61d9a/volumes"
Jul 16 22:25:24.074735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.074703    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075008    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075018    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075031    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075036    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075045    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075050    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075062    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.075088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.075360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075116    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075123    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.075360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075130    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:24.075360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.075135    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:24.079423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.079404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:24.081906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.081885    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:25:24.082024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.081952    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:25:24.085051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.085020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:24.177908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.177877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9\" (UID: \"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:24.279201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.279163    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9\" (UID: \"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:24.279814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.279793    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9\" (UID: \"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:24.391426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.391394    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:24.515864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.515841    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:24.518012 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:25:24.517977    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2fba4c9_e1d1_4285_9160_c0df0f1e45a3.slice/crio-a3389f87b24d31b29a1001fb1fb70b8b68e8c1dd3b3a14ee29ddc1d5bd7e06f5 WatchSource:0}: Error finding container a3389f87b24d31b29a1001fb1fb70b8b68e8c1dd3b3a14ee29ddc1d5bd7e06f5: Status 404 returned error can't find the container with id a3389f87b24d31b29a1001fb1fb70b8b68e8c1dd3b3a14ee29ddc1d5bd7e06f5
Jul 16 22:25:24.852083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:24.852002    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" event={"ID":"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3","Type":"ContainerStarted","Data":"a3389f87b24d31b29a1001fb1fb70b8b68e8c1dd3b3a14ee29ddc1d5bd7e06f5"}
Jul 16 22:25:25.857716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:25.857669    2581 generic.go:356] "Generic (PLEG): container finished" podID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerID="487ff9a1c76a400054ced646f31d681a9374dd8ebd2be2574c2433dbd4bb6c8e" exitCode=7
Jul 16 22:25:25.858129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:25.857748    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" event={"ID":"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3","Type":"ContainerDied","Data":"487ff9a1c76a400054ced646f31d681a9374dd8ebd2be2574c2433dbd4bb6c8e"}
Jul 16 22:25:25.858129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:25.857921    2581 scope.go:117] "RemoveContainer" containerID="487ff9a1c76a400054ced646f31d681a9374dd8ebd2be2574c2433dbd4bb6c8e"
Jul 16 22:25:26.863664 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:26.863624    2581 generic.go:356] "Generic (PLEG): container finished" podID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerID="eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d" exitCode=7
Jul 16 22:25:26.864064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:26.863705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" event={"ID":"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3","Type":"ContainerDied","Data":"eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d"}
Jul 16 22:25:26.864064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:26.863760    2581 scope.go:117] "RemoveContainer" containerID="487ff9a1c76a400054ced646f31d681a9374dd8ebd2be2574c2433dbd4bb6c8e"
Jul 16 22:25:26.864064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:26.863951    2581 scope.go:117] "RemoveContainer" containerID="eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d"
Jul 16 22:25:26.864197 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:26.864174    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9_odh-ai-gateway-infra(d2fba4c9-e1d1-4285-9160-c0df0f1e45a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3"
Jul 16 22:25:27.869387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:27.869362    2581 scope.go:117] "RemoveContainer" containerID="eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d"
Jul 16 22:25:27.869760 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:27.869550    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9_odh-ai-gateway-infra(d2fba4c9-e1d1-4285-9160-c0df0f1e45a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3"
Jul 16 22:25:40.155351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:40.155308    2581 scope.go:117] "RemoveContainer" containerID="eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d"
Jul 16 22:25:40.927354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:40.927315    2581 generic.go:356] "Generic (PLEG): container finished" podID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerID="0b2728babb7256f4b29eae8bfedaaf3497b6a2b59e771e31af1b5fe7d95b8345" exitCode=7
Jul 16 22:25:40.927509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:40.927391    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" event={"ID":"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3","Type":"ContainerDied","Data":"0b2728babb7256f4b29eae8bfedaaf3497b6a2b59e771e31af1b5fe7d95b8345"}
Jul 16 22:25:40.927509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:40.927453    2581 scope.go:117] "RemoveContainer" containerID="eee85958e802ab84d8a1feaa33595d567f0cc87cd7f0613d03c8e45cf878097d"
Jul 16 22:25:40.927761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:40.927740    2581 scope.go:117] "RemoveContainer" containerID="0b2728babb7256f4b29eae8bfedaaf3497b6a2b59e771e31af1b5fe7d95b8345"
Jul 16 22:25:40.927941 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:40.927923    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9_odh-ai-gateway-infra(d2fba4c9-e1d1-4285-9160-c0df0f1e45a3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3"
Jul 16 22:25:41.958317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:41.958282    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:42.082885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.082862    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:42.123886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.123857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca\") pod \"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3\" (UID: \"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3\") "
Jul 16 22:25:42.124178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.124154    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" (UID: "d2fba4c9-e1d1-4285-9160-c0df0f1e45a3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:42.225208 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.225126    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:25:42.938665 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.938634    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"
Jul 16 22:25:42.938848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.938639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9" event={"ID":"d2fba4c9-e1d1-4285-9160-c0df0f1e45a3","Type":"ContainerDied","Data":"a3389f87b24d31b29a1001fb1fb70b8b68e8c1dd3b3a14ee29ddc1d5bd7e06f5"}
Jul 16 22:25:42.938848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.938797    2581 scope.go:117] "RemoveContainer" containerID="0b2728babb7256f4b29eae8bfedaaf3497b6a2b59e771e31af1b5fe7d95b8345"
Jul 16 22:25:42.968259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.968217    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:42.976771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:42.976746    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ql7v9"]
Jul 16 22:25:43.160051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:43.160017    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" path="/var/lib/kubelet/pods/d2fba4c9-e1d1-4285-9160-c0df0f1e45a3/volumes"
Jul 16 22:25:44.954884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.954813    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955160    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955174    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955185    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955191    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955208    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955213    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955223    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955247    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955297    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955304    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:44.955526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955312    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="032f45b2-772e-4067-837d-ac7aae291cc2" containerName="revoke-keys"
Jul 16 22:25:44.955526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.955317    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3eadb2d-0415-48c9-acc6-8604c3d61d9a" containerName="revoke-keys"
Jul 16 22:25:44.959690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.959672    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:25:44.962553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.962526    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:25:44.962688 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.962622    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:25:44.971931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:44.971909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:25:45.053734 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.053705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl\" (UID: \"9631ab51-589b-4617-8057-6e6c1a090aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:25:45.154587 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.154555    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl\" (UID: \"9631ab51-589b-4617-8057-6e6c1a090aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:25:45.155113 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.155096    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl\" (UID: \"9631ab51-589b-4617-8057-6e6c1a090aa8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:25:45.269244 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.269155    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:25:45.396575 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.396476    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:25:45.399190 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:25:45.399161    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9631ab51_589b_4617_8057_6e6c1a090aa8.slice/crio-0f262fcc6f3df7aa287d94e5427d501799fb48bceb40f49983856c93ff5e704b WatchSource:0}: Error finding container 0f262fcc6f3df7aa287d94e5427d501799fb48bceb40f49983856c93ff5e704b: Status 404 returned error can't find the container with id 0f262fcc6f3df7aa287d94e5427d501799fb48bceb40f49983856c93ff5e704b
Jul 16 22:25:45.956629 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.956594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerStarted","Data":"2f2de998ef7d07ba0ca19074ba1e4a35fb58ea83cfb0e3af8715b8a8aae5fb01"}
Jul 16 22:25:45.956971 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.956648    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerStarted","Data":"0f262fcc6f3df7aa287d94e5427d501799fb48bceb40f49983856c93ff5e704b"}
Jul 16 22:25:45.977828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:45.977783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" podStartSLOduration=1.977767364 podStartE2EDuration="1.977767364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:45.97685222 +0000 UTC m=+2333.306292122" watchObservedRunningTime="2026-07-16 22:25:45.977767364 +0000 UTC m=+2333.307207265"
Jul 16 22:25:46.479486 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.479448    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:25:46.479783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.479770    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:46.479834 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.479784    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:46.479877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.479834    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2fba4c9-e1d1-4285-9160-c0df0f1e45a3" containerName="revoke-keys"
Jul 16 22:25:46.482732 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.482716    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:25:46.490991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.490967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:25:46.565610 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.565581    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b\" (UID: \"2d0aa924-ff7e-45dc-91c0-d9232e802e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:25:46.666329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.666301    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b\" (UID: \"2d0aa924-ff7e-45dc-91c0-d9232e802e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:25:46.666881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.666863    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b\" (UID: \"2d0aa924-ff7e-45dc-91c0-d9232e802e06\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:25:46.794020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.793951    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:25:46.920276 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.920252    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:25:46.921818 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:25:46.921784    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d0aa924_ff7e_45dc_91c0_d9232e802e06.slice/crio-2c04e97d99f08c3cc426857e8393269d5f082dae06d4837c89d1460c8c17f443 WatchSource:0}: Error finding container 2c04e97d99f08c3cc426857e8393269d5f082dae06d4837c89d1460c8c17f443: Status 404 returned error can't find the container with id 2c04e97d99f08c3cc426857e8393269d5f082dae06d4837c89d1460c8c17f443
Jul 16 22:25:46.962716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.962687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" event={"ID":"2d0aa924-ff7e-45dc-91c0-d9232e802e06","Type":"ContainerStarted","Data":"2c04e97d99f08c3cc426857e8393269d5f082dae06d4837c89d1460c8c17f443"}
Jul 16 22:25:46.964222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.964199    2581 generic.go:356] "Generic (PLEG): container finished" podID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerID="2f2de998ef7d07ba0ca19074ba1e4a35fb58ea83cfb0e3af8715b8a8aae5fb01" exitCode=6
Jul 16 22:25:46.964332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.964268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerDied","Data":"2f2de998ef7d07ba0ca19074ba1e4a35fb58ea83cfb0e3af8715b8a8aae5fb01"}
Jul 16 22:25:46.964449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:46.964435    2581 scope.go:117] "RemoveContainer" containerID="2f2de998ef7d07ba0ca19074ba1e4a35fb58ea83cfb0e3af8715b8a8aae5fb01"
Jul 16 22:25:47.969537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.969498    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerID="9cdb577f8302e9c13cd3d254f35153bc087cb4274d599eba96e427e0e9a5a256" exitCode=6
Jul 16 22:25:47.969951 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.969578    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" event={"ID":"2d0aa924-ff7e-45dc-91c0-d9232e802e06","Type":"ContainerDied","Data":"9cdb577f8302e9c13cd3d254f35153bc087cb4274d599eba96e427e0e9a5a256"}
Jul 16 22:25:47.969951 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.969748    2581 scope.go:117] "RemoveContainer" containerID="9cdb577f8302e9c13cd3d254f35153bc087cb4274d599eba96e427e0e9a5a256"
Jul 16 22:25:47.971639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.971617    2581 generic.go:356] "Generic (PLEG): container finished" podID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerID="a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84" exitCode=6
Jul 16 22:25:47.971751 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.971665    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerDied","Data":"a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84"}
Jul 16 22:25:47.971751 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.971716    2581 scope.go:117] "RemoveContainer" containerID="2f2de998ef7d07ba0ca19074ba1e4a35fb58ea83cfb0e3af8715b8a8aae5fb01"
Jul 16 22:25:47.971865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:47.971849    2581 scope.go:117] "RemoveContainer" containerID="a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84"
Jul 16 22:25:47.972074 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:47.972051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl_odh-ai-gateway-infra(9631ab51-589b-4617-8057-6e6c1a090aa8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8"
Jul 16 22:25:48.978248 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:48.978198    2581 scope.go:117] "RemoveContainer" containerID="a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84"
Jul 16 22:25:48.978844 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:48.978448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl_odh-ai-gateway-infra(9631ab51-589b-4617-8057-6e6c1a090aa8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8"
Jul 16 22:25:48.979740 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:48.979716    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerID="875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8" exitCode=6
Jul 16 22:25:48.979822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:48.979797    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" event={"ID":"2d0aa924-ff7e-45dc-91c0-d9232e802e06","Type":"ContainerDied","Data":"875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8"}
Jul 16 22:25:48.979865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:48.979843    2581 scope.go:117] "RemoveContainer" containerID="9cdb577f8302e9c13cd3d254f35153bc087cb4274d599eba96e427e0e9a5a256"
Jul 16 22:25:48.979936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:48.979919    2581 scope.go:117] "RemoveContainer" containerID="875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8"
Jul 16 22:25:48.980113 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:48.980092    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b_odh-ai-gateway-infra(2d0aa924-ff7e-45dc-91c0-d9232e802e06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06"
Jul 16 22:25:49.986029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:25:49.986003    2581 scope.go:117] "RemoveContainer" containerID="875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8"
Jul 16 22:25:49.986449 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:25:49.986206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b_odh-ai-gateway-infra(2d0aa924-ff7e-45dc-91c0-d9232e802e06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06"
Jul 16 22:26:01.154806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:01.154772    2581 scope.go:117] "RemoveContainer" containerID="a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84"
Jul 16 22:26:01.155377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:01.154890    2581 scope.go:117] "RemoveContainer" containerID="875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8"
Jul 16 22:26:02.042674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.042593    2581 generic.go:356] "Generic (PLEG): container finished" podID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerID="0ac64b1681e476507225aa7f5e6cff78aab009f44a19e8bb6d4018e7adeea549" exitCode=6
Jul 16 22:26:02.042674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.042664    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerDied","Data":"0ac64b1681e476507225aa7f5e6cff78aab009f44a19e8bb6d4018e7adeea549"}
Jul 16 22:26:02.042891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.042721    2581 scope.go:117] "RemoveContainer" containerID="a90541918e04897275bec5c9cda847bc985c56a73339922ffacb9ed543426f84"
Jul 16 22:26:02.043019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.043006    2581 scope.go:117] "RemoveContainer" containerID="0ac64b1681e476507225aa7f5e6cff78aab009f44a19e8bb6d4018e7adeea549"
Jul 16 22:26:02.043269 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:02.043242    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl_odh-ai-gateway-infra(9631ab51-589b-4617-8057-6e6c1a090aa8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8"
Jul 16 22:26:02.044449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.044430    2581 generic.go:356] "Generic (PLEG): container finished" podID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerID="e827e89ecbce146d409d923984561a81a799e09c58d527bd7c6acbd9377c05b5" exitCode=6
Jul 16 22:26:02.044562 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.044504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" event={"ID":"2d0aa924-ff7e-45dc-91c0-d9232e802e06","Type":"ContainerDied","Data":"e827e89ecbce146d409d923984561a81a799e09c58d527bd7c6acbd9377c05b5"}
Jul 16 22:26:02.044758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.044744    2581 scope.go:117] "RemoveContainer" containerID="e827e89ecbce146d409d923984561a81a799e09c58d527bd7c6acbd9377c05b5"
Jul 16 22:26:02.044910 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:02.044895    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b_odh-ai-gateway-infra(2d0aa924-ff7e-45dc-91c0-d9232e802e06)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06"
Jul 16 22:26:02.053550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:02.053537    2581 scope.go:117] "RemoveContainer" containerID="875904431a7320edc487420c6ca1387603c8f8cd78ba407ca9f79ddf5fd410f8"
Jul 16 22:26:03.081384 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.081355    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:26:03.111411 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.111371    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:26:03.217186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.217160    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:26:03.261173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.261151    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:26:03.306100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.306073    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca\") pod \"9631ab51-589b-4617-8057-6e6c1a090aa8\" (UID: \"9631ab51-589b-4617-8057-6e6c1a090aa8\") "
Jul 16 22:26:03.306421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.306399    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9631ab51-589b-4617-8057-6e6c1a090aa8" (UID: "9631ab51-589b-4617-8057-6e6c1a090aa8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:03.406772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.406740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca\") pod \"2d0aa924-ff7e-45dc-91c0-d9232e802e06\" (UID: \"2d0aa924-ff7e-45dc-91c0-d9232e802e06\") "
Jul 16 22:26:03.406933 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.406901    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9631ab51-589b-4617-8057-6e6c1a090aa8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:26:03.407064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.407043    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d0aa924-ff7e-45dc-91c0-d9232e802e06" (UID: "2d0aa924-ff7e-45dc-91c0-d9232e802e06"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:03.507549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:03.507515    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d0aa924-ff7e-45dc-91c0-d9232e802e06-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:26:04.056721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.056693    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"
Jul 16 22:26:04.056721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.056707    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl" event={"ID":"9631ab51-589b-4617-8057-6e6c1a090aa8","Type":"ContainerDied","Data":"0f262fcc6f3df7aa287d94e5427d501799fb48bceb40f49983856c93ff5e704b"}
Jul 16 22:26:04.056970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.056776    2581 scope.go:117] "RemoveContainer" containerID="0ac64b1681e476507225aa7f5e6cff78aab009f44a19e8bb6d4018e7adeea549"
Jul 16 22:26:04.058059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.058039    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"
Jul 16 22:26:04.058185 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.058067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b" event={"ID":"2d0aa924-ff7e-45dc-91c0-d9232e802e06","Type":"ContainerDied","Data":"2c04e97d99f08c3cc426857e8393269d5f082dae06d4837c89d1460c8c17f443"}
Jul 16 22:26:04.064940 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.064921    2581 scope.go:117] "RemoveContainer" containerID="e827e89ecbce146d409d923984561a81a799e09c58d527bd7c6acbd9377c05b5"
Jul 16 22:26:04.082286 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.082261    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:26:04.087090 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.087066    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w6f8b"]
Jul 16 22:26:04.098338 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.098315    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:26:04.103926 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:04.103907    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrpbl"]
Jul 16 22:26:05.159819 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:05.159789    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" path="/var/lib/kubelet/pods/2d0aa924-ff7e-45dc-91c0-d9232e802e06/volumes"
Jul 16 22:26:05.160186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:05.160101    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" path="/var/lib/kubelet/pods/9631ab51-589b-4617-8057-6e6c1a090aa8/volumes"
Jul 16 22:26:14.129662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.129625    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:14.130099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130066    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130083    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130098    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130107    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130134    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130142    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130160    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130168    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130267    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130277    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130287    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.130301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130297    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130307    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:14.130707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.130316    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:14.134841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.134820    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:14.137531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.137509    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:26:14.137603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.137531    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:26:14.143483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.143460    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:14.295047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.295013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k\" (UID: \"48891715-1197-435d-88dd-f9b5605e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:14.396204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.396119    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k\" (UID: \"48891715-1197-435d-88dd-f9b5605e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:14.396725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.396705    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k\" (UID: \"48891715-1197-435d-88dd-f9b5605e0cad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:14.446187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.446164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:14.576707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:14.576683    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:14.579570 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:26:14.579535    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48891715_1197_435d_88dd_f9b5605e0cad.slice/crio-1cec70371a356c0cd5ac245859697b5ce47b423db0d3de9a6a93d2ecd735d688 WatchSource:0}: Error finding container 1cec70371a356c0cd5ac245859697b5ce47b423db0d3de9a6a93d2ecd735d688: Status 404 returned error can't find the container with id 1cec70371a356c0cd5ac245859697b5ce47b423db0d3de9a6a93d2ecd735d688
Jul 16 22:26:15.106057 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:15.106023    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerStarted","Data":"1c80d358043a01b82d0cea5f8e6118d7b2d0cc15993dec31ce585c8087586620"}
Jul 16 22:26:15.106212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:15.106096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerStarted","Data":"1cec70371a356c0cd5ac245859697b5ce47b423db0d3de9a6a93d2ecd735d688"}
Jul 16 22:26:15.122149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:15.122103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" podStartSLOduration=1.122091128 podStartE2EDuration="1.122091128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:15.120408485 +0000 UTC m=+2362.449848389" watchObservedRunningTime="2026-07-16 22:26:15.122091128 +0000 UTC m=+2362.451531029"
Jul 16 22:26:16.112141 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:16.112106    2581 generic.go:356] "Generic (PLEG): container finished" podID="48891715-1197-435d-88dd-f9b5605e0cad" containerID="1c80d358043a01b82d0cea5f8e6118d7b2d0cc15993dec31ce585c8087586620" exitCode=7
Jul 16 22:26:16.112621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:16.112182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerDied","Data":"1c80d358043a01b82d0cea5f8e6118d7b2d0cc15993dec31ce585c8087586620"}
Jul 16 22:26:16.112621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:16.112406    2581 scope.go:117] "RemoveContainer" containerID="1c80d358043a01b82d0cea5f8e6118d7b2d0cc15993dec31ce585c8087586620"
Jul 16 22:26:17.117930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:17.117896    2581 generic.go:356] "Generic (PLEG): container finished" podID="48891715-1197-435d-88dd-f9b5605e0cad" containerID="4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a" exitCode=7
Jul 16 22:26:17.118377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:17.117987    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerDied","Data":"4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a"}
Jul 16 22:26:17.118377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:17.118041    2581 scope.go:117] "RemoveContainer" containerID="1c80d358043a01b82d0cea5f8e6118d7b2d0cc15993dec31ce585c8087586620"
Jul 16 22:26:17.118377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:17.118166    2581 scope.go:117] "RemoveContainer" containerID="4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a"
Jul 16 22:26:17.118535 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:17.118411    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k_odh-ai-gateway-infra(48891715-1197-435d-88dd-f9b5605e0cad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" podUID="48891715-1197-435d-88dd-f9b5605e0cad"
Jul 16 22:26:18.124076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:18.124048    2581 scope.go:117] "RemoveContainer" containerID="4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a"
Jul 16 22:26:18.124465 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:18.124268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k_odh-ai-gateway-infra(48891715-1197-435d-88dd-f9b5605e0cad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" podUID="48891715-1197-435d-88dd-f9b5605e0cad"
Jul 16 22:26:33.157303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:33.157275    2581 scope.go:117] "RemoveContainer" containerID="4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a"
Jul 16 22:26:34.193377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:34.193339    2581 generic.go:356] "Generic (PLEG): container finished" podID="48891715-1197-435d-88dd-f9b5605e0cad" containerID="5db51b2608876e7cb0200f91ccbf3759f30c22cbeb9a60d385e2ff6860a376c7" exitCode=7
Jul 16 22:26:34.193820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:34.193411    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerDied","Data":"5db51b2608876e7cb0200f91ccbf3759f30c22cbeb9a60d385e2ff6860a376c7"}
Jul 16 22:26:34.193820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:34.193470    2581 scope.go:117] "RemoveContainer" containerID="4f55964aa3e851443f5a3e163a8bffbb3baeeedcade0915b09d1c404fd20b75a"
Jul 16 22:26:34.193820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:34.193757    2581 scope.go:117] "RemoveContainer" containerID="5db51b2608876e7cb0200f91ccbf3759f30c22cbeb9a60d385e2ff6860a376c7"
Jul 16 22:26:34.193998 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:34.193977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k_odh-ai-gateway-infra(48891715-1197-435d-88dd-f9b5605e0cad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" podUID="48891715-1197-435d-88dd-f9b5605e0cad"
Jul 16 22:26:35.005466 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.005430    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:35.005777 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.005762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:35.005823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.005780    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d0aa924-ff7e-45dc-91c0-d9232e802e06" containerName="revoke-keys"
Jul 16 22:26:35.005823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.005803    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:35.005823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.005808    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9631ab51-589b-4617-8057-6e6c1a090aa8" containerName="revoke-keys"
Jul 16 22:26:35.010047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.010029    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:35.019078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.019035    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:35.068585 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.068557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f\" (UID: \"0e20ae5d-febe-449c-81fb-d90cd44b29ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:35.169720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.169690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f\" (UID: \"0e20ae5d-febe-449c-81fb-d90cd44b29ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:35.170256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.170217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f\" (UID: \"0e20ae5d-febe-449c-81fb-d90cd44b29ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:35.228600 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.228564    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:35.321360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.321335    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:35.352503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.352473    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:35.463507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.463481    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:35.464206 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:26:35.464187    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e20ae5d_febe_449c_81fb_d90cd44b29ef.slice/crio-ca617f4cc471164eefc00f17919bcd2f7c01030e064b81f923bb6cc16e83dbe4 WatchSource:0}: Error finding container ca617f4cc471164eefc00f17919bcd2f7c01030e064b81f923bb6cc16e83dbe4: Status 404 returned error can't find the container with id ca617f4cc471164eefc00f17919bcd2f7c01030e064b81f923bb6cc16e83dbe4
Jul 16 22:26:35.472154 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.472127    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca\") pod \"48891715-1197-435d-88dd-f9b5605e0cad\" (UID: \"48891715-1197-435d-88dd-f9b5605e0cad\") "
Jul 16 22:26:35.472453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.472430    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "48891715-1197-435d-88dd-f9b5605e0cad" (UID: "48891715-1197-435d-88dd-f9b5605e0cad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:35.578288 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:35.573639    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/48891715-1197-435d-88dd-f9b5605e0cad-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:26:36.204670 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.204630    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerID="8870e797c3e9cb5ea0880b80c425fb15f78bd2f9863826d519d4764792220d9c" exitCode=6
Jul 16 22:26:36.204798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.204694    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" event={"ID":"0e20ae5d-febe-449c-81fb-d90cd44b29ef","Type":"ContainerDied","Data":"8870e797c3e9cb5ea0880b80c425fb15f78bd2f9863826d519d4764792220d9c"}
Jul 16 22:26:36.204798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.204743    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" event={"ID":"0e20ae5d-febe-449c-81fb-d90cd44b29ef","Type":"ContainerStarted","Data":"ca617f4cc471164eefc00f17919bcd2f7c01030e064b81f923bb6cc16e83dbe4"}
Jul 16 22:26:36.204931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.204917    2581 scope.go:117] "RemoveContainer" containerID="8870e797c3e9cb5ea0880b80c425fb15f78bd2f9863826d519d4764792220d9c"
Jul 16 22:26:36.205972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.205944    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k" event={"ID":"48891715-1197-435d-88dd-f9b5605e0cad","Type":"ContainerDied","Data":"1cec70371a356c0cd5ac245859697b5ce47b423db0d3de9a6a93d2ecd735d688"}
Jul 16 22:26:36.206066 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.205984    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"
Jul 16 22:26:36.206107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.205988    2581 scope.go:117] "RemoveContainer" containerID="5db51b2608876e7cb0200f91ccbf3759f30c22cbeb9a60d385e2ff6860a376c7"
Jul 16 22:26:36.264602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.264576    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:36.267494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.267474    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-stp8k"]
Jul 16 22:26:36.555612 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555578    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:36.555900 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555888    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.555946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.555946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555909    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.555946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555915    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.555946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.555946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555935    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.556095 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.556095 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.555996    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.556095 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.556001    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="48891715-1197-435d-88dd-f9b5605e0cad" containerName="revoke-keys"
Jul 16 22:26:36.558828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.558810    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:36.588342 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.588315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:36.681641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.681571    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6\" (UID: \"f7aa883e-d61a-4a96-acd3-d22dab2dd045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:36.782672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.782648    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6\" (UID: \"f7aa883e-d61a-4a96-acd3-d22dab2dd045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:36.783212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.783192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6\" (UID: \"f7aa883e-d61a-4a96-acd3-d22dab2dd045\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:36.868516 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.868480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:36.997072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:36.997041    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:36.999370 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:26:36.999340    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7aa883e_d61a_4a96_acd3_d22dab2dd045.slice/crio-f823ef6274c1dc24b9b9d5ed67e526c76c327999e4bf7622990c07ef8de2012b WatchSource:0}: Error finding container f823ef6274c1dc24b9b9d5ed67e526c76c327999e4bf7622990c07ef8de2012b: Status 404 returned error can't find the container with id f823ef6274c1dc24b9b9d5ed67e526c76c327999e4bf7622990c07ef8de2012b
Jul 16 22:26:37.160079 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.160048    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="48891715-1197-435d-88dd-f9b5605e0cad" path="/var/lib/kubelet/pods/48891715-1197-435d-88dd-f9b5605e0cad/volumes"
Jul 16 22:26:37.211313 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.211237    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" event={"ID":"f7aa883e-d61a-4a96-acd3-d22dab2dd045","Type":"ContainerStarted","Data":"f823ef6274c1dc24b9b9d5ed67e526c76c327999e4bf7622990c07ef8de2012b"}
Jul 16 22:26:37.213651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.213624    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerID="daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a" exitCode=6
Jul 16 22:26:37.213774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.213662    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" event={"ID":"0e20ae5d-febe-449c-81fb-d90cd44b29ef","Type":"ContainerDied","Data":"daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a"}
Jul 16 22:26:37.213774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.213695    2581 scope.go:117] "RemoveContainer" containerID="8870e797c3e9cb5ea0880b80c425fb15f78bd2f9863826d519d4764792220d9c"
Jul 16 22:26:37.213891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:37.213870    2581 scope.go:117] "RemoveContainer" containerID="daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a"
Jul 16 22:26:37.214077 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:37.214058    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f_odh-ai-gateway-infra(0e20ae5d-febe-449c-81fb-d90cd44b29ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef"
Jul 16 22:26:38.219480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:38.219448    2581 scope.go:117] "RemoveContainer" containerID="daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a"
Jul 16 22:26:38.219896 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:38.219690    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f_odh-ai-gateway-infra(0e20ae5d-febe-449c-81fb-d90cd44b29ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef"
Jul 16 22:26:38.220648 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:38.220622    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerID="c971ca9184d5efc1ff1045dba939f729e48514c5bdcdec8c90d45bf0b6d11b2a" exitCode=6
Jul 16 22:26:38.220760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:38.220702    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" event={"ID":"f7aa883e-d61a-4a96-acd3-d22dab2dd045","Type":"ContainerDied","Data":"c971ca9184d5efc1ff1045dba939f729e48514c5bdcdec8c90d45bf0b6d11b2a"}
Jul 16 22:26:38.220845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:38.220832    2581 scope.go:117] "RemoveContainer" containerID="c971ca9184d5efc1ff1045dba939f729e48514c5bdcdec8c90d45bf0b6d11b2a"
Jul 16 22:26:39.227157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:39.227066    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerID="690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec" exitCode=6
Jul 16 22:26:39.227569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:39.227154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" event={"ID":"f7aa883e-d61a-4a96-acd3-d22dab2dd045","Type":"ContainerDied","Data":"690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec"}
Jul 16 22:26:39.227569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:39.227213    2581 scope.go:117] "RemoveContainer" containerID="c971ca9184d5efc1ff1045dba939f729e48514c5bdcdec8c90d45bf0b6d11b2a"
Jul 16 22:26:39.227569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:39.227424    2581 scope.go:117] "RemoveContainer" containerID="690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec"
Jul 16 22:26:39.227709 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:39.227648    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6_odh-ai-gateway-infra(f7aa883e-d61a-4a96-acd3-d22dab2dd045)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045"
Jul 16 22:26:40.233397 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:40.233368    2581 scope.go:117] "RemoveContainer" containerID="690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec"
Jul 16 22:26:40.233777 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:40.233552    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6_odh-ai-gateway-infra(f7aa883e-d61a-4a96-acd3-d22dab2dd045)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045"
Jul 16 22:26:49.154829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:49.154798    2581 scope.go:117] "RemoveContainer" containerID="daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a"
Jul 16 22:26:50.277506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:50.277475    2581 generic.go:356] "Generic (PLEG): container finished" podID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerID="625cfd0e1568831b91926189433015c25e66806d0663c1e26ff69ab7517214c1" exitCode=6
Jul 16 22:26:50.278037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:50.277550    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" event={"ID":"0e20ae5d-febe-449c-81fb-d90cd44b29ef","Type":"ContainerDied","Data":"625cfd0e1568831b91926189433015c25e66806d0663c1e26ff69ab7517214c1"}
Jul 16 22:26:50.278037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:50.277611    2581 scope.go:117] "RemoveContainer" containerID="daf22a5dee6c4a1072783cd9306178db3aa7f282e2bdaf9d829dd173bfa74e2a"
Jul 16 22:26:50.278037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:50.277956    2581 scope.go:117] "RemoveContainer" containerID="625cfd0e1568831b91926189433015c25e66806d0663c1e26ff69ab7517214c1"
Jul 16 22:26:50.278205 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:50.278185    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f_odh-ai-gateway-infra(0e20ae5d-febe-449c-81fb-d90cd44b29ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef"
Jul 16 22:26:51.316419 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:51.316387    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:51.450474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:51.450446    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:51.507535 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:51.507504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca\") pod \"0e20ae5d-febe-449c-81fb-d90cd44b29ef\" (UID: \"0e20ae5d-febe-449c-81fb-d90cd44b29ef\") "
Jul 16 22:26:51.507849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:51.507827    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e20ae5d-febe-449c-81fb-d90cd44b29ef" (UID: "0e20ae5d-febe-449c-81fb-d90cd44b29ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:51.608938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:51.608901    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e20ae5d-febe-449c-81fb-d90cd44b29ef-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:26:52.289331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:52.289295    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"
Jul 16 22:26:52.289331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:52.289319    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f" event={"ID":"0e20ae5d-febe-449c-81fb-d90cd44b29ef","Type":"ContainerDied","Data":"ca617f4cc471164eefc00f17919bcd2f7c01030e064b81f923bb6cc16e83dbe4"}
Jul 16 22:26:52.289574 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:52.289384    2581 scope.go:117] "RemoveContainer" containerID="625cfd0e1568831b91926189433015c25e66806d0663c1e26ff69ab7517214c1"
Jul 16 22:26:52.309596 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:52.309572    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:52.315472 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:52.315451    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-xth5f"]
Jul 16 22:26:53.161178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:53.161137    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" path="/var/lib/kubelet/pods/0e20ae5d-febe-449c-81fb-d90cd44b29ef/volumes"
Jul 16 22:26:54.463271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:54.463245    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:26:54.492714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:54.492690    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:26:55.155142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:55.155113    2581 scope.go:117] "RemoveContainer" containerID="690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec"
Jul 16 22:26:56.309836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:56.309744    2581 generic.go:356] "Generic (PLEG): container finished" podID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerID="e5fbb214557e0cec185352b11072f81f3424b7600e3485b185e453d35ac72f29" exitCode=6
Jul 16 22:26:56.309836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:56.309822    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" event={"ID":"f7aa883e-d61a-4a96-acd3-d22dab2dd045","Type":"ContainerDied","Data":"e5fbb214557e0cec185352b11072f81f3424b7600e3485b185e453d35ac72f29"}
Jul 16 22:26:56.310328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:56.309895    2581 scope.go:117] "RemoveContainer" containerID="690c16122c2f34fca658d51487e7eb0ec5eb2ba3acfed6ba0bc344311ab9adec"
Jul 16 22:26:56.310328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:56.310193    2581 scope.go:117] "RemoveContainer" containerID="e5fbb214557e0cec185352b11072f81f3424b7600e3485b185e453d35ac72f29"
Jul 16 22:26:56.310451 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:26:56.310426    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6_odh-ai-gateway-infra(f7aa883e-d61a-4a96-acd3-d22dab2dd045)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045"
Jul 16 22:26:57.337192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:57.337158    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:57.480349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:57.480326    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:57.558917 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:57.558886    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca\") pod \"f7aa883e-d61a-4a96-acd3-d22dab2dd045\" (UID: \"f7aa883e-d61a-4a96-acd3-d22dab2dd045\") "
Jul 16 22:26:57.559274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:57.559221    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f7aa883e-d61a-4a96-acd3-d22dab2dd045" (UID: "f7aa883e-d61a-4a96-acd3-d22dab2dd045"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:57.659800 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:57.659721    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f7aa883e-d61a-4a96-acd3-d22dab2dd045-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:26:58.321838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:58.321809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"
Jul 16 22:26:58.321838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:58.321828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6" event={"ID":"f7aa883e-d61a-4a96-acd3-d22dab2dd045","Type":"ContainerDied","Data":"f823ef6274c1dc24b9b9d5ed67e526c76c327999e4bf7622990c07ef8de2012b"}
Jul 16 22:26:58.322103 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:58.321887    2581 scope.go:117] "RemoveContainer" containerID="e5fbb214557e0cec185352b11072f81f3424b7600e3485b185e453d35ac72f29"
Jul 16 22:26:58.344781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:58.344759    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:58.352284 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:58.352261    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-rwxq6"]
Jul 16 22:26:59.160305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:26:59.160271    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" path="/var/lib/kubelet/pods/f7aa883e-d61a-4a96-acd3-d22dab2dd045/volumes"
Jul 16 22:27:14.185718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.185681    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186070    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186089    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186112    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186119    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186136    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186146    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186155    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186183    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186282    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186293    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:14.186590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186302    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186310    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.186590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.186318    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:14.189407 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.189393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:14.192034 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.192014    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:27:14.192159 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.192015    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:27:14.197017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.196996    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:14.296500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.296431    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z\" (UID: \"e9ac031c-0187-4410-abbe-10625c6e68da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:14.397724 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.397683    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z\" (UID: \"e9ac031c-0187-4410-abbe-10625c6e68da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:14.398331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.398306    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z\" (UID: \"e9ac031c-0187-4410-abbe-10625c6e68da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:14.500762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.500692    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:14.623609 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:14.623582    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:14.625592 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:27:14.625563    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9ac031c_0187_4410_abbe_10625c6e68da.slice/crio-11a970270e994208b646f8285c85df18b607298365145e25a9bbb0c3f27b9867 WatchSource:0}: Error finding container 11a970270e994208b646f8285c85df18b607298365145e25a9bbb0c3f27b9867: Status 404 returned error can't find the container with id 11a970270e994208b646f8285c85df18b607298365145e25a9bbb0c3f27b9867
Jul 16 22:27:15.399685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:15.399654    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9ac031c-0187-4410-abbe-10625c6e68da" containerID="f63a63f7a7f68990fa1e5ff6c5c648e0d4d0ae9429305df4d0b7d429bf795453" exitCode=7
Jul 16 22:27:15.400019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:15.399741    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerDied","Data":"f63a63f7a7f68990fa1e5ff6c5c648e0d4d0ae9429305df4d0b7d429bf795453"}
Jul 16 22:27:15.400019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:15.399791    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerStarted","Data":"11a970270e994208b646f8285c85df18b607298365145e25a9bbb0c3f27b9867"}
Jul 16 22:27:15.400019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:15.399903    2581 scope.go:117] "RemoveContainer" containerID="f63a63f7a7f68990fa1e5ff6c5c648e0d4d0ae9429305df4d0b7d429bf795453"
Jul 16 22:27:16.406343 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:16.406307    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9ac031c-0187-4410-abbe-10625c6e68da" containerID="07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b" exitCode=7
Jul 16 22:27:16.406836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:16.406392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerDied","Data":"07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b"}
Jul 16 22:27:16.406836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:16.406449    2581 scope.go:117] "RemoveContainer" containerID="f63a63f7a7f68990fa1e5ff6c5c648e0d4d0ae9429305df4d0b7d429bf795453"
Jul 16 22:27:16.406836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:16.406662    2581 scope.go:117] "RemoveContainer" containerID="07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b"
Jul 16 22:27:16.406976 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:16.406876    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z_odh-ai-gateway-infra(e9ac031c-0187-4410-abbe-10625c6e68da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" podUID="e9ac031c-0187-4410-abbe-10625c6e68da"
Jul 16 22:27:17.412858 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:17.412830    2581 scope.go:117] "RemoveContainer" containerID="07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b"
Jul 16 22:27:17.413250 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:17.413009    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z_odh-ai-gateway-infra(e9ac031c-0187-4410-abbe-10625c6e68da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" podUID="e9ac031c-0187-4410-abbe-10625c6e68da"
Jul 16 22:27:25.051597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.051567    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:25.051966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.051900    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:25.051966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.051911    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e20ae5d-febe-449c-81fb-d90cd44b29ef" containerName="revoke-keys"
Jul 16 22:27:25.051966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.051962    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f7aa883e-d61a-4a96-acd3-d22dab2dd045" containerName="revoke-keys"
Jul 16 22:27:25.055974 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.055954    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:25.065080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.065054    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:25.071479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.071455    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m\" (UID: \"fb544e2b-bc6a-4a2c-98a3-952a822713c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:25.172746 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.172718    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m\" (UID: \"fb544e2b-bc6a-4a2c-98a3-952a822713c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:25.173379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.173354    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m\" (UID: \"fb544e2b-bc6a-4a2c-98a3-952a822713c9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:25.366978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.366945    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:25.513591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:25.513563    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:25.516643 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:27:25.516598    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb544e2b_bc6a_4a2c_98a3_952a822713c9.slice/crio-62bc8b7af66ba1521dbdd68ccea5d13c650868a596d3558fb31137694afc7a1d WatchSource:0}: Error finding container 62bc8b7af66ba1521dbdd68ccea5d13c650868a596d3558fb31137694afc7a1d: Status 404 returned error can't find the container with id 62bc8b7af66ba1521dbdd68ccea5d13c650868a596d3558fb31137694afc7a1d
Jul 16 22:27:26.451372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:26.451272    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerID="e2852f54ea5cc07e4172bec20e52a6646cd8735a2afd36a7bafd9c96ad706a82" exitCode=6
Jul 16 22:27:26.451720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:26.451366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerDied","Data":"e2852f54ea5cc07e4172bec20e52a6646cd8735a2afd36a7bafd9c96ad706a82"}
Jul 16 22:27:26.451720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:26.451426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerStarted","Data":"62bc8b7af66ba1521dbdd68ccea5d13c650868a596d3558fb31137694afc7a1d"}
Jul 16 22:27:26.451720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:26.451557    2581 scope.go:117] "RemoveContainer" containerID="e2852f54ea5cc07e4172bec20e52a6646cd8735a2afd36a7bafd9c96ad706a82"
Jul 16 22:27:27.457375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:27.457342    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerID="f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b" exitCode=6
Jul 16 22:27:27.457803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:27.457389    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerDied","Data":"f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b"}
Jul 16 22:27:27.457803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:27.457436    2581 scope.go:117] "RemoveContainer" containerID="e2852f54ea5cc07e4172bec20e52a6646cd8735a2afd36a7bafd9c96ad706a82"
Jul 16 22:27:27.457803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:27.457594    2581 scope.go:117] "RemoveContainer" containerID="f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b"
Jul 16 22:27:27.457803 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:27.457780    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m_odh-ai-gateway-infra(fb544e2b-bc6a-4a2c-98a3-952a822713c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9"
Jul 16 22:27:28.464152 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:28.464124    2581 scope.go:117] "RemoveContainer" containerID="f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b"
Jul 16 22:27:28.464517 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:28.464296    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m_odh-ai-gateway-infra(fb544e2b-bc6a-4a2c-98a3-952a822713c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9"
Jul 16 22:27:30.154899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:30.154866    2581 scope.go:117] "RemoveContainer" containerID="07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b"
Jul 16 22:27:30.474593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:30.474507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerStarted","Data":"dfbce912a6ccaf1c5ddd6d80cfaa89641bd678d9073fe60052ec48e48154aa4e"}
Jul 16 22:27:30.501727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:30.501678    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" podStartSLOduration=16.501664337 podStartE2EDuration="16.501664337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:30.500460429 +0000 UTC m=+2437.829900331" watchObservedRunningTime="2026-07-16 22:27:30.501664337 +0000 UTC m=+2437.831104241"
Jul 16 22:27:31.481347 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:31.481314    2581 generic.go:356] "Generic (PLEG): container finished" podID="e9ac031c-0187-4410-abbe-10625c6e68da" containerID="dfbce912a6ccaf1c5ddd6d80cfaa89641bd678d9073fe60052ec48e48154aa4e" exitCode=7
Jul 16 22:27:31.481758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:31.481385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerDied","Data":"dfbce912a6ccaf1c5ddd6d80cfaa89641bd678d9073fe60052ec48e48154aa4e"}
Jul 16 22:27:31.481758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:31.481443    2581 scope.go:117] "RemoveContainer" containerID="07eb487effa79c5faf1f887f06f7ae864a7cd469ffa2389777fda708c895333b"
Jul 16 22:27:31.481758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:31.481703    2581 scope.go:117] "RemoveContainer" containerID="dfbce912a6ccaf1c5ddd6d80cfaa89641bd678d9073fe60052ec48e48154aa4e"
Jul 16 22:27:31.481899 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:31.481878    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z_odh-ai-gateway-infra(e9ac031c-0187-4410-abbe-10625c6e68da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" podUID="e9ac031c-0187-4410-abbe-10625c6e68da"
Jul 16 22:27:31.517305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:31.517274    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:32.615621 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:32.615599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:32.629673 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:32.629650    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca\") pod \"e9ac031c-0187-4410-abbe-10625c6e68da\" (UID: \"e9ac031c-0187-4410-abbe-10625c6e68da\") "
Jul 16 22:27:32.629970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:32.629951    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e9ac031c-0187-4410-abbe-10625c6e68da" (UID: "e9ac031c-0187-4410-abbe-10625c6e68da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:32.730771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:32.730736    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e9ac031c-0187-4410-abbe-10625c6e68da-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:33.492012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:33.491921    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"
Jul 16 22:27:33.492012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:33.491941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z" event={"ID":"e9ac031c-0187-4410-abbe-10625c6e68da","Type":"ContainerDied","Data":"11a970270e994208b646f8285c85df18b607298365145e25a9bbb0c3f27b9867"}
Jul 16 22:27:33.492012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:33.492003    2581 scope.go:117] "RemoveContainer" containerID="dfbce912a6ccaf1c5ddd6d80cfaa89641bd678d9073fe60052ec48e48154aa4e"
Jul 16 22:27:33.509327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:33.509296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:33.514002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:33.513979    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-52x9z"]
Jul 16 22:27:35.159973 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:35.159939    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" path="/var/lib/kubelet/pods/e9ac031c-0187-4410-abbe-10625c6e68da/volumes"
Jul 16 22:27:36.596619 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596587    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596903    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596921    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596926    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596933    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.596979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596939    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.597177 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.596989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.597177 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.597000    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:36.600993 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.600977    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:36.609615 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.609588    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:36.660555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.660530    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c\" (UID: \"707dd2ad-b20a-41f3-aab6-8bd49ff7b715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:36.761149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.761125    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c\" (UID: \"707dd2ad-b20a-41f3-aab6-8bd49ff7b715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:36.761773 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.761753    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c\" (UID: \"707dd2ad-b20a-41f3-aab6-8bd49ff7b715\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:36.911939 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:36.911875    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:37.246738 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:37.246708    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:37.249213 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:27:37.249188    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod707dd2ad_b20a_41f3_aab6_8bd49ff7b715.slice/crio-1893a85bb9a727e182ac0d941605dab69772f57ce922fd86aa603aca023dd479 WatchSource:0}: Error finding container 1893a85bb9a727e182ac0d941605dab69772f57ce922fd86aa603aca023dd479: Status 404 returned error can't find the container with id 1893a85bb9a727e182ac0d941605dab69772f57ce922fd86aa603aca023dd479
Jul 16 22:27:37.511496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:37.511419    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerStarted","Data":"1893a85bb9a727e182ac0d941605dab69772f57ce922fd86aa603aca023dd479"}
Jul 16 22:27:38.517102 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:38.517067    2581 generic.go:356] "Generic (PLEG): container finished" podID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerID="da7801b72ec253729d771cfe2b79245dc620c7669adc1048ee0559c124a229c5" exitCode=6
Jul 16 22:27:38.517514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:38.517134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerDied","Data":"da7801b72ec253729d771cfe2b79245dc620c7669adc1048ee0559c124a229c5"}
Jul 16 22:27:38.517514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:38.517305    2581 scope.go:117] "RemoveContainer" containerID="da7801b72ec253729d771cfe2b79245dc620c7669adc1048ee0559c124a229c5"
Jul 16 22:27:39.523280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:39.523171    2581 generic.go:356] "Generic (PLEG): container finished" podID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerID="6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8" exitCode=6
Jul 16 22:27:39.523280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:39.523263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerDied","Data":"6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8"}
Jul 16 22:27:39.523741 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:39.523323    2581 scope.go:117] "RemoveContainer" containerID="da7801b72ec253729d771cfe2b79245dc620c7669adc1048ee0559c124a229c5"
Jul 16 22:27:39.523741 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:39.523508    2581 scope.go:117] "RemoveContainer" containerID="6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8"
Jul 16 22:27:39.523741 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:39.523719    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c_odh-ai-gateway-infra(707dd2ad-b20a-41f3-aab6-8bd49ff7b715)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715"
Jul 16 22:27:40.530201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:40.530175    2581 scope.go:117] "RemoveContainer" containerID="6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8"
Jul 16 22:27:40.530713 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:40.530415    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c_odh-ai-gateway-infra(707dd2ad-b20a-41f3-aab6-8bd49ff7b715)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715"
Jul 16 22:27:44.146614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:44.146581    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:27:44.147116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:44.146846    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy" containerID="cri-o://7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9" gracePeriod=30
Jul 16 22:27:44.154986 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:44.154962    2581 scope.go:117] "RemoveContainer" containerID="f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b"
Jul 16 22:27:44.549236 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:44.549191    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerStarted","Data":"0d943c85b5f211d27c678d0b1003d8dc4217cd05070e51089a9921fd652de7a5"}
Jul 16 22:27:44.578342 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:44.578297    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" podStartSLOduration=19.57828269 podStartE2EDuration="19.57828269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:44.576521276 +0000 UTC m=+2451.905961177" watchObservedRunningTime="2026-07-16 22:27:44.57828269 +0000 UTC m=+2451.907722570"
Jul 16 22:27:45.555778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:45.555748    2581 generic.go:356] "Generic (PLEG): container finished" podID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerID="0d943c85b5f211d27c678d0b1003d8dc4217cd05070e51089a9921fd652de7a5" exitCode=6
Jul 16 22:27:45.556196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:45.555818    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerDied","Data":"0d943c85b5f211d27c678d0b1003d8dc4217cd05070e51089a9921fd652de7a5"}
Jul 16 22:27:45.556196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:45.555881    2581 scope.go:117] "RemoveContainer" containerID="f1ea73c5b050222f91adc223467214a9a8320bff8bf2750b622cd7b50c698d3b"
Jul 16 22:27:45.556196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:45.556115    2581 scope.go:117] "RemoveContainer" containerID="0d943c85b5f211d27c678d0b1003d8dc4217cd05070e51089a9921fd652de7a5"
Jul 16 22:27:45.556373 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:45.556350    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m_odh-ai-gateway-infra(fb544e2b-bc6a-4a2c-98a3-952a822713c9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9"
Jul 16 22:27:45.587748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:45.587721    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:46.694328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:46.694305    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:46.745861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:46.745837    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca\") pod \"fb544e2b-bc6a-4a2c-98a3-952a822713c9\" (UID: \"fb544e2b-bc6a-4a2c-98a3-952a822713c9\") "
Jul 16 22:27:46.746187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:46.746164    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fb544e2b-bc6a-4a2c-98a3-952a822713c9" (UID: "fb544e2b-bc6a-4a2c-98a3-952a822713c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:46.847142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:46.847117    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb544e2b-bc6a-4a2c-98a3-952a822713c9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:46.931391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:46.931351    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:27:47.567446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:47.567364    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m" event={"ID":"fb544e2b-bc6a-4a2c-98a3-952a822713c9","Type":"ContainerDied","Data":"62bc8b7af66ba1521dbdd68ccea5d13c650868a596d3558fb31137694afc7a1d"}
Jul 16 22:27:47.567446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:47.567390    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"
Jul 16 22:27:47.567446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:47.567427    2581 scope.go:117] "RemoveContainer" containerID="0d943c85b5f211d27c678d0b1003d8dc4217cd05070e51089a9921fd652de7a5"
Jul 16 22:27:47.583691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:47.583665    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:47.589631 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:47.589608    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2vz5m"]
Jul 16 22:27:49.160401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.160373    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" path="/var/lib/kubelet/pods/fb544e2b-bc6a-4a2c-98a3-952a822713c9/volumes"
Jul 16 22:27:49.397128 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.397103    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:27:49.469996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.469970    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470030    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bz7rh\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470060    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470120    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470147    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470425    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470468    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470498    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:49.470611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470528    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470558    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data\") pod \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\" (UID: \"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5\") "
Jul 16 22:27:49.470934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470717    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:49.470934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470865    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.470934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470885    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.471093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.470991    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:49.471093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.471074    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:49.471793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.471761    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data" (OuterVolumeSpecName: "istio-data") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:49.472529 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.472505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:27:49.472766 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.472744    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token" (OuterVolumeSpecName: "istio-token") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:49.472945 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.472928    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:27:49.473469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.473449    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh" (OuterVolumeSpecName: "kube-api-access-bz7rh") pod "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" (UID: "99b84cbf-5d43-47b4-bc12-67d8f63e4aa5"). InnerVolumeSpecName "kube-api-access-bz7rh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:27:49.571723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571698    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571720    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571729    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571737    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571745    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bz7rh\" (UniqueName: \"kubernetes.io/projected/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-kube-api-access-bz7rh\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571753    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.571869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.571760    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:49.578707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.578677    2581 generic.go:356] "Generic (PLEG): container finished" podID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerID="7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9" exitCode=0
Jul 16 22:27:49.578806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.578735    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"
Jul 16 22:27:49.578806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.578768    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerDied","Data":"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"}
Jul 16 22:27:49.578880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.578823    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5" event={"ID":"99b84cbf-5d43-47b4-bc12-67d8f63e4aa5","Type":"ContainerDied","Data":"9fb7edd239e067be797270590e2901f7578213233b9322f8076efed14b491465"}
Jul 16 22:27:49.578880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.578845    2581 scope.go:117] "RemoveContainer" containerID="7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"
Jul 16 22:27:49.587402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.587383    2581 scope.go:117] "RemoveContainer" containerID="a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"
Jul 16 22:27:49.594517 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.594500    2581 scope.go:117] "RemoveContainer" containerID="7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"
Jul 16 22:27:49.594775 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:49.594758    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9\": container with ID starting with 7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9 not found: ID does not exist" containerID="7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"
Jul 16 22:27:49.594822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.594783    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9"} err="failed to get container status \"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9\": rpc error: code = NotFound desc = could not find container \"7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9\": container with ID starting with 7fd1f54f64a2eed20e7984af61998f44172d580e08941364dc460a2a4028e5c9 not found: ID does not exist"
Jul 16 22:27:49.594822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.594799    2581 scope.go:117] "RemoveContainer" containerID="a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"
Jul 16 22:27:49.594995 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:49.594981    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036\": container with ID starting with a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036 not found: ID does not exist" containerID="a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"
Jul 16 22:27:49.595040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.594997    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036"} err="failed to get container status \"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036\": rpc error: code = NotFound desc = could not find container \"a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036\": container with ID starting with a70542aefbf6e1c3a8072a62e27b121b56bf70b06e76f86b3f3fddab89779036 not found: ID does not exist"
Jul 16 22:27:49.602218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.602199    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:27:49.609403 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:49.609382    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-24cdbda3-openshift-default-76c7fb4dd9-7bct5"]
Jul 16 22:27:51.155534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:51.155504    2581 scope.go:117] "RemoveContainer" containerID="6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8"
Jul 16 22:27:51.160873 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:51.160849    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" path="/var/lib/kubelet/pods/99b84cbf-5d43-47b4-bc12-67d8f63e4aa5/volumes"
Jul 16 22:27:51.589472 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:51.589440    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerStarted","Data":"e4bc71884586bb74467a1b3d397a67a8040a13f54067ae49def4824a685fec16"}
Jul 16 22:27:51.605956 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:51.605916    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" podStartSLOduration=15.605902404 podStartE2EDuration="15.605902404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:51.605252672 +0000 UTC m=+2458.934692575" watchObservedRunningTime="2026-07-16 22:27:51.605902404 +0000 UTC m=+2458.935342305"
Jul 16 22:27:52.595689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:52.595658    2581 generic.go:356] "Generic (PLEG): container finished" podID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerID="e4bc71884586bb74467a1b3d397a67a8040a13f54067ae49def4824a685fec16" exitCode=6
Jul 16 22:27:52.596129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:52.595704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerDied","Data":"e4bc71884586bb74467a1b3d397a67a8040a13f54067ae49def4824a685fec16"}
Jul 16 22:27:52.596129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:52.595751    2581 scope.go:117] "RemoveContainer" containerID="6d44cb883f1520adebc648f5a260ed39f9339e8c1515eccef167fb5b8bfb64e8"
Jul 16 22:27:52.596129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:52.596069    2581 scope.go:117] "RemoveContainer" containerID="e4bc71884586bb74467a1b3d397a67a8040a13f54067ae49def4824a685fec16"
Jul 16 22:27:52.596331 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:27:52.596294    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c_odh-ai-gateway-infra(707dd2ad-b20a-41f3-aab6-8bd49ff7b715)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715"
Jul 16 22:27:52.620892 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:52.620869    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:53.735085 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:53.735060    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:53.807848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:53.807818    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca\") pod \"707dd2ad-b20a-41f3-aab6-8bd49ff7b715\" (UID: \"707dd2ad-b20a-41f3-aab6-8bd49ff7b715\") "
Jul 16 22:27:53.808168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:53.808143    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "707dd2ad-b20a-41f3-aab6-8bd49ff7b715" (UID: "707dd2ad-b20a-41f3-aab6-8bd49ff7b715"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:53.908932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:53.908845    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/707dd2ad-b20a-41f3-aab6-8bd49ff7b715-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:27:54.607320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:54.607291    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"
Jul 16 22:27:54.607320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:54.607303    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c" event={"ID":"707dd2ad-b20a-41f3-aab6-8bd49ff7b715","Type":"ContainerDied","Data":"1893a85bb9a727e182ac0d941605dab69772f57ce922fd86aa603aca023dd479"}
Jul 16 22:27:54.607583 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:54.607378    2581 scope.go:117] "RemoveContainer" containerID="e4bc71884586bb74467a1b3d397a67a8040a13f54067ae49def4824a685fec16"
Jul 16 22:27:54.631613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:54.631586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:54.638611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:54.638591    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-thw5c"]
Jul 16 22:27:55.160085 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:55.160050    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" path="/var/lib/kubelet/pods/707dd2ad-b20a-41f3-aab6-8bd49ff7b715/volumes"
Jul 16 22:27:57.109881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.109850    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:27:57.110279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110264    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110281    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110305    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110314    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110324    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110341    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110349    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110364    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110371    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110387    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110399    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110406    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110478    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110488    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110500    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="99b84cbf-5d43-47b4-bc12-67d8f63e4aa5" containerName="istio-proxy"
Jul 16 22:27:57.110507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110508    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.111070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110517    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e9ac031c-0187-4410-abbe-10625c6e68da" containerName="revoke-keys"
Jul 16 22:27:57.111070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110527    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:27:57.111070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.110534    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:27:57.117726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.117707    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.120451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.120432    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a6237863-openshift-default-dockercfg-wrr7w\""
Jul 16 22:27:57.129386 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.129366    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:27:57.236403 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236368    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236427    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236448    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236479    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236555    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8d4pw\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236601    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236645    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.236735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.236671    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337302    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337346    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8d4pw\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337366    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337393    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337410    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337455    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337586    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337667    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337788    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337842    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.337968 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.337885    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.338066 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.338037    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.338300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.338254    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.339908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.339887    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.340042 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.340024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.345893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.345861    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.346830 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.346807    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8d4pw\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw\") pod \"e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") " pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.428619 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.428548    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:57.562315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.562291    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:27:57.564088 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:27:57.564065    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24e42674_0b4d_47be_8eeb_c9a9919720da.slice/crio-a5c874529b4182f75a4d36375e5bbdb44d98494e188a6ac1ad7c03af780f4a64 WatchSource:0}: Error finding container a5c874529b4182f75a4d36375e5bbdb44d98494e188a6ac1ad7c03af780f4a64: Status 404 returned error can't find the container with id a5c874529b4182f75a4d36375e5bbdb44d98494e188a6ac1ad7c03af780f4a64
Jul 16 22:27:57.566409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.566377    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:27:57.566497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.566440    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:27:57.566497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.566466    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:27:57.620903 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.620876    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerStarted","Data":"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"}
Jul 16 22:27:57.621025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.620925    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerStarted","Data":"a5c874529b4182f75a4d36375e5bbdb44d98494e188a6ac1ad7c03af780f4a64"}
Jul 16 22:27:57.639838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:57.639783    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podStartSLOduration=0.639763683 podStartE2EDuration="639.763683ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:57.638918661 +0000 UTC m=+2464.968358564" watchObservedRunningTime="2026-07-16 22:27:57.639763683 +0000 UTC m=+2464.969203586"
Jul 16 22:27:58.428843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:58.428811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:58.430281 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:58.430259    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:27:58.430337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:58.430315    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:27:59.433523 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:59.433491    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:59.631121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:59.631092    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:27:59.632202 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:27:59.632184    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:28:02.662529 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.662497    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:02.663002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.662936    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:28:02.663002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.662952    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:28:02.663109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.663042    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="707dd2ad-b20a-41f3-aab6-8bd49ff7b715" containerName="revoke-keys"
Jul 16 22:28:02.663109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.663058    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb544e2b-bc6a-4a2c-98a3-952a822713c9" containerName="revoke-keys"
Jul 16 22:28:02.665943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.665924    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:02.668516 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.668497    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:28:02.668634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.668534    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:02.674147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.674122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:02.785178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.785143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7qt2j\" (UID: \"fe361e2e-c762-41a0-a32b-add6e7d409b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:02.886496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.886464    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7qt2j\" (UID: \"fe361e2e-c762-41a0-a32b-add6e7d409b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:02.887161 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.887141    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7qt2j\" (UID: \"fe361e2e-c762-41a0-a32b-add6e7d409b3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:02.976781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:02.976656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:03.107360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:03.107324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:03.110214 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:28:03.110185    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe361e2e_c762_41a0_a32b_add6e7d409b3.slice/crio-64d671e2afcd881490d5cb588b1e2c4d38f242d5aeca438b178a18156b82eb76 WatchSource:0}: Error finding container 64d671e2afcd881490d5cb588b1e2c4d38f242d5aeca438b178a18156b82eb76: Status 404 returned error can't find the container with id 64d671e2afcd881490d5cb588b1e2c4d38f242d5aeca438b178a18156b82eb76
Jul 16 22:28:03.650796 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:03.650697    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerStarted","Data":"57c6ac5076508174590160fe0f592d1bbb77939284e9872f158b9563872bca38"}
Jul 16 22:28:03.650796 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:03.650764    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerStarted","Data":"64d671e2afcd881490d5cb588b1e2c4d38f242d5aeca438b178a18156b82eb76"}
Jul 16 22:28:03.668421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:03.668376    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" podStartSLOduration=1.668361996 podStartE2EDuration="1.668361996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:03.666546978 +0000 UTC m=+2470.995986880" watchObservedRunningTime="2026-07-16 22:28:03.668361996 +0000 UTC m=+2470.997801898"
Jul 16 22:28:04.236409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.236378    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:04.239630 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.239613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:04.257079 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.257053    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:04.399718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.399689    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tj729\" (UID: \"66d1ff08-3a73-4ade-8fef-914373dfaaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:04.500672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.500580    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tj729\" (UID: \"66d1ff08-3a73-4ade-8fef-914373dfaaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:04.501775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.501736    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tj729\" (UID: \"66d1ff08-3a73-4ade-8fef-914373dfaaa2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:04.549257 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.549192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:04.659037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.659006    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerID="57c6ac5076508174590160fe0f592d1bbb77939284e9872f158b9563872bca38" exitCode=7
Jul 16 22:28:04.659191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.659086    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerDied","Data":"57c6ac5076508174590160fe0f592d1bbb77939284e9872f158b9563872bca38"}
Jul 16 22:28:04.659375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.659357    2581 scope.go:117] "RemoveContainer" containerID="57c6ac5076508174590160fe0f592d1bbb77939284e9872f158b9563872bca38"
Jul 16 22:28:04.704600 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:04.704577    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:04.707024 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:28:04.706998    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d1ff08_3a73_4ade_8fef_914373dfaaa2.slice/crio-1a32765635e18a0f85cf31f2e4024f85001a294348cdd06d79a9958107192561 WatchSource:0}: Error finding container 1a32765635e18a0f85cf31f2e4024f85001a294348cdd06d79a9958107192561: Status 404 returned error can't find the container with id 1a32765635e18a0f85cf31f2e4024f85001a294348cdd06d79a9958107192561
Jul 16 22:28:05.664644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.664609    2581 generic.go:356] "Generic (PLEG): container finished" podID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerID="4e6b176183e5c09071a9fc7e871f5cd1dac00b6059e7e729e44d3487ead96e95" exitCode=6
Jul 16 22:28:05.664824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.664695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerDied","Data":"4e6b176183e5c09071a9fc7e871f5cd1dac00b6059e7e729e44d3487ead96e95"}
Jul 16 22:28:05.664824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.664744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerStarted","Data":"1a32765635e18a0f85cf31f2e4024f85001a294348cdd06d79a9958107192561"}
Jul 16 22:28:05.664931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.664842    2581 scope.go:117] "RemoveContainer" containerID="4e6b176183e5c09071a9fc7e871f5cd1dac00b6059e7e729e44d3487ead96e95"
Jul 16 22:28:05.666526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.666501    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerID="06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c" exitCode=7
Jul 16 22:28:05.666642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.666544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerDied","Data":"06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c"}
Jul 16 22:28:05.666642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.666581    2581 scope.go:117] "RemoveContainer" containerID="57c6ac5076508174590160fe0f592d1bbb77939284e9872f158b9563872bca38"
Jul 16 22:28:05.666781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:05.666766    2581 scope.go:117] "RemoveContainer" containerID="06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c"
Jul 16 22:28:05.667018 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:05.666954    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7qt2j_odh-ai-gateway-infra(fe361e2e-c762-41a0-a32b-add6e7d409b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3"
Jul 16 22:28:06.672383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.672349    2581 scope.go:117] "RemoveContainer" containerID="06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c"
Jul 16 22:28:06.672839 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:06.672570    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7qt2j_odh-ai-gateway-infra(fe361e2e-c762-41a0-a32b-add6e7d409b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3"
Jul 16 22:28:06.673863 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.673840    2581 generic.go:356] "Generic (PLEG): container finished" podID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerID="1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a" exitCode=6
Jul 16 22:28:06.673981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.673884    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerDied","Data":"1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a"}
Jul 16 22:28:06.673981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.673924    2581 scope.go:117] "RemoveContainer" containerID="4e6b176183e5c09071a9fc7e871f5cd1dac00b6059e7e729e44d3487ead96e95"
Jul 16 22:28:06.674111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.674094    2581 scope.go:117] "RemoveContainer" containerID="1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a"
Jul 16 22:28:06.674291 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:06.674273    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tj729_odh-ai-gateway-infra(66d1ff08-3a73-4ade-8fef-914373dfaaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2"
Jul 16 22:28:06.809292 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:06.809249    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:28:07.680382 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:07.680352    2581 scope.go:117] "RemoveContainer" containerID="1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a"
Jul 16 22:28:07.680920 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:07.680540    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tj729_odh-ai-gateway-infra(66d1ff08-3a73-4ade-8fef-914373dfaaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2"
Jul 16 22:28:14.632566 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.632533    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:28:14.632989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.632588    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:28:14.712795 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.712718    2581 generic.go:356] "Generic (PLEG): container finished" podID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerID="cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c" exitCode=0
Jul 16 22:28:14.712795 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.712765    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerDied","Data":"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"}
Jul 16 22:28:14.713148 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.713129    2581 scope.go:117] "RemoveContainer" containerID="cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"
Jul 16 22:28:14.714123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.714093    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:28:14.714183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.714162    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:28:14.714241 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:14.714216    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:28:15.719050 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:15.719013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerStarted","Data":"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"}
Jul 16 22:28:16.429056 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:16.429016    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:28:16.429313 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:16.429291    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:28:16.429366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:16.429339    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:28:17.429375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:17.429336    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:28:17.429769 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:17.429402    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:28:18.429756 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:18.429720    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:28:18.430150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:18.429791    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:28:19.158037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:19.158004    2581 scope.go:117] "RemoveContainer" containerID="06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c"
Jul 16 22:28:19.429069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:19.428983    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused" start-of-body=
Jul 16 22:28:19.429069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:19.429038    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.59:15021/healthz/ready\": dial tcp 10.133.0.59:15021: connect: connection refused"
Jul 16 22:28:19.737351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:19.737261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerStarted","Data":"19b833555a6a74de0f15e58c62b3c0c8b9de4acc430ad7b2c10f9a07a5068892"}
Jul 16 22:28:20.155046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.155020    2581 scope.go:117] "RemoveContainer" containerID="1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a"
Jul 16 22:28:20.432727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.432660    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:28:20.432949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.432927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:28:20.433761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.433740    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:28:20.743898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.743818    2581 generic.go:356] "Generic (PLEG): container finished" podID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerID="19b833555a6a74de0f15e58c62b3c0c8b9de4acc430ad7b2c10f9a07a5068892" exitCode=7
Jul 16 22:28:20.744378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.743898    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerDied","Data":"19b833555a6a74de0f15e58c62b3c0c8b9de4acc430ad7b2c10f9a07a5068892"}
Jul 16 22:28:20.744378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.743959    2581 scope.go:117] "RemoveContainer" containerID="06f3329329a6f7a6e75e142c2390de80131155f65fa8819dbda2baceee8d924c"
Jul 16 22:28:20.744378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.744288    2581 scope.go:117] "RemoveContainer" containerID="19b833555a6a74de0f15e58c62b3c0c8b9de4acc430ad7b2c10f9a07a5068892"
Jul 16 22:28:20.744557 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:20.744498    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7qt2j_odh-ai-gateway-infra(fe361e2e-c762-41a0-a32b-add6e7d409b3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3"
Jul 16 22:28:20.746054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.746030    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerStarted","Data":"51df99a97d31a75cf984c7b58cada8b8cd7b6452776028a468d50def0fdf235d"}
Jul 16 22:28:20.769170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.769145    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:20.777994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:20.777956    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" podStartSLOduration=16.77794292 podStartE2EDuration="16.77794292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:20.777580644 +0000 UTC m=+2488.107020548" watchObservedRunningTime="2026-07-16 22:28:20.77794292 +0000 UTC m=+2488.107382822"
Jul 16 22:28:21.752467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.752433    2581 generic.go:356] "Generic (PLEG): container finished" podID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerID="51df99a97d31a75cf984c7b58cada8b8cd7b6452776028a468d50def0fdf235d" exitCode=6
Jul 16 22:28:21.752898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.752509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerDied","Data":"51df99a97d31a75cf984c7b58cada8b8cd7b6452776028a468d50def0fdf235d"}
Jul 16 22:28:21.752898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.752573    2581 scope.go:117] "RemoveContainer" containerID="1b657f5eac3b6b911d12512ac8388eccfd1f29293079b67e817da29848e35a2a"
Jul 16 22:28:21.752898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.752863    2581 scope.go:117] "RemoveContainer" containerID="51df99a97d31a75cf984c7b58cada8b8cd7b6452776028a468d50def0fdf235d"
Jul 16 22:28:21.753100 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:21.753075    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tj729_odh-ai-gateway-infra(66d1ff08-3a73-4ade-8fef-914373dfaaa2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2"
Jul 16 22:28:21.798153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.798122    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:21.889442 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.889418    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:21.950907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.950877    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca\") pod \"fe361e2e-c762-41a0-a32b-add6e7d409b3\" (UID: \"fe361e2e-c762-41a0-a32b-add6e7d409b3\") "
Jul 16 22:28:21.951296 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:21.951272    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fe361e2e-c762-41a0-a32b-add6e7d409b3" (UID: "fe361e2e-c762-41a0-a32b-add6e7d409b3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:22.051780 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.051695    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fe361e2e-c762-41a0-a32b-add6e7d409b3-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:28:22.761606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.761575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j" event={"ID":"fe361e2e-c762-41a0-a32b-add6e7d409b3","Type":"ContainerDied","Data":"64d671e2afcd881490d5cb588b1e2c4d38f242d5aeca438b178a18156b82eb76"}
Jul 16 22:28:22.761606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.761599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"
Jul 16 22:28:22.762002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.761653    2581 scope.go:117] "RemoveContainer" containerID="19b833555a6a74de0f15e58c62b3c0c8b9de4acc430ad7b2c10f9a07a5068892"
Jul 16 22:28:22.789515 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.789484    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:22.792146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.792124    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7qt2j"]
Jul 16 22:28:22.891339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.891314    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:22.958454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.958426    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca\") pod \"66d1ff08-3a73-4ade-8fef-914373dfaaa2\" (UID: \"66d1ff08-3a73-4ade-8fef-914373dfaaa2\") "
Jul 16 22:28:22.958745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:22.958721    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "66d1ff08-3a73-4ade-8fef-914373dfaaa2" (UID: "66d1ff08-3a73-4ade-8fef-914373dfaaa2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:23.059359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.059284    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/66d1ff08-3a73-4ade-8fef-914373dfaaa2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:28:23.160825 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.160794    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" path="/var/lib/kubelet/pods/fe361e2e-c762-41a0-a32b-add6e7d409b3/volumes"
Jul 16 22:28:23.767647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.767617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"
Jul 16 22:28:23.768133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.767660    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729" event={"ID":"66d1ff08-3a73-4ade-8fef-914373dfaaa2","Type":"ContainerDied","Data":"1a32765635e18a0f85cf31f2e4024f85001a294348cdd06d79a9958107192561"}
Jul 16 22:28:23.768133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.767725    2581 scope.go:117] "RemoveContainer" containerID="51df99a97d31a75cf984c7b58cada8b8cd7b6452776028a468d50def0fdf235d"
Jul 16 22:28:23.783273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.783250    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:23.786954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:23.786936    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tj729"]
Jul 16 22:28:25.086821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.086786    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087108    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087120    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087150    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087159    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087164    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087171    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087176    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087182    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087186    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087266    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:25.087512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087280    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.087512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.087290    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:25.091465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.091445    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:25.093994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.093972    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:28:25.094106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.094019    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:25.097418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.097398    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:25.160847 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.160813    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" path="/var/lib/kubelet/pods/66d1ff08-3a73-4ade-8fef-914373dfaaa2/volumes"
Jul 16 22:28:25.174500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.174474    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf\" (UID: \"f3d8d081-7d8a-4585-9bcf-06c0545484cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:25.275538 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.275506    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf\" (UID: \"f3d8d081-7d8a-4585-9bcf-06c0545484cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:25.276086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.276067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf\" (UID: \"f3d8d081-7d8a-4585-9bcf-06c0545484cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:25.402784 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.402698    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:25.529391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.529350    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:25.531541 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:28:25.531512    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d8d081_7d8a_4585_9bcf_06c0545484cb.slice/crio-2b683f10f62f7d746c7bca544b4589139a6cc449faf6f982f86bb50362cfa024 WatchSource:0}: Error finding container 2b683f10f62f7d746c7bca544b4589139a6cc449faf6f982f86bb50362cfa024: Status 404 returned error can't find the container with id 2b683f10f62f7d746c7bca544b4589139a6cc449faf6f982f86bb50362cfa024
Jul 16 22:28:25.780845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.780816    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerStarted","Data":"757eb2708435b02de44f9280471bd6d0873a0ec277defd3f15b206f8fbe72e6c"}
Jul 16 22:28:25.780969 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.780869    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerStarted","Data":"2b683f10f62f7d746c7bca544b4589139a6cc449faf6f982f86bb50362cfa024"}
Jul 16 22:28:25.805223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:25.805176    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" podStartSLOduration=0.805161751 podStartE2EDuration="805.161751ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:25.80410843 +0000 UTC m=+2493.133548333" watchObservedRunningTime="2026-07-16 22:28:25.805161751 +0000 UTC m=+2493.134601652"
Jul 16 22:28:26.634652 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.634617    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:26.635040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.634930    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:26.635040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.634941    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fe361e2e-c762-41a0-a32b-add6e7d409b3" containerName="revoke-keys"
Jul 16 22:28:26.635040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.635010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="66d1ff08-3a73-4ade-8fef-914373dfaaa2" containerName="revoke-keys"
Jul 16 22:28:26.638050 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.638033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:26.645339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.645196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:26.686704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.686678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9\" (UID: \"34442e52-c511-4b5d-b9ee-228691dfbc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:26.787267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.787214    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9\" (UID: \"34442e52-c511-4b5d-b9ee-228691dfbc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:26.787267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.787257    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerDied","Data":"757eb2708435b02de44f9280471bd6d0873a0ec277defd3f15b206f8fbe72e6c"}
Jul 16 22:28:26.787541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.787217    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerID="757eb2708435b02de44f9280471bd6d0873a0ec277defd3f15b206f8fbe72e6c" exitCode=6
Jul 16 22:28:26.787541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.787485    2581 scope.go:117] "RemoveContainer" containerID="757eb2708435b02de44f9280471bd6d0873a0ec277defd3f15b206f8fbe72e6c"
Jul 16 22:28:26.788182 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.787869    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9\" (UID: \"34442e52-c511-4b5d-b9ee-228691dfbc86\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:26.950027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:26.949964    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:27.079674 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.079646    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:27.081207 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:28:27.081179    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34442e52_c511_4b5d_b9ee_228691dfbc86.slice/crio-2217d58033bef58853a6673f711aea29f3ef49d1b60065c4af3e5f30bdaf6338 WatchSource:0}: Error finding container 2217d58033bef58853a6673f711aea29f3ef49d1b60065c4af3e5f30bdaf6338: Status 404 returned error can't find the container with id 2217d58033bef58853a6673f711aea29f3ef49d1b60065c4af3e5f30bdaf6338
Jul 16 22:28:27.793546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.793508    2581 generic.go:356] "Generic (PLEG): container finished" podID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerID="75b523b7884e263c0071bf26a1c40efd7ce26252cc62963cc22782b979be3a91" exitCode=6
Jul 16 22:28:27.793991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.793564    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" event={"ID":"34442e52-c511-4b5d-b9ee-228691dfbc86","Type":"ContainerDied","Data":"75b523b7884e263c0071bf26a1c40efd7ce26252cc62963cc22782b979be3a91"}
Jul 16 22:28:27.793991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.793617    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" event={"ID":"34442e52-c511-4b5d-b9ee-228691dfbc86","Type":"ContainerStarted","Data":"2217d58033bef58853a6673f711aea29f3ef49d1b60065c4af3e5f30bdaf6338"}
Jul 16 22:28:27.793991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.793828    2581 scope.go:117] "RemoveContainer" containerID="75b523b7884e263c0071bf26a1c40efd7ce26252cc62963cc22782b979be3a91"
Jul 16 22:28:27.795449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.795426    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerID="eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19" exitCode=6
Jul 16 22:28:27.795559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.795506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerDied","Data":"eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19"}
Jul 16 22:28:27.795559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.795552    2581 scope.go:117] "RemoveContainer" containerID="757eb2708435b02de44f9280471bd6d0873a0ec277defd3f15b206f8fbe72e6c"
Jul 16 22:28:27.795749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:27.795724    2581 scope.go:117] "RemoveContainer" containerID="eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19"
Jul 16 22:28:27.796006 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:27.795925    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf_odh-ai-gateway-infra(f3d8d081-7d8a-4585-9bcf-06c0545484cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb"
Jul 16 22:28:28.801316 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:28.801208    2581 scope.go:117] "RemoveContainer" containerID="eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19"
Jul 16 22:28:28.801768 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:28.801475    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf_odh-ai-gateway-infra(f3d8d081-7d8a-4585-9bcf-06c0545484cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb"
Jul 16 22:28:28.802765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:28.802741    2581 generic.go:356] "Generic (PLEG): container finished" podID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerID="e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197" exitCode=6
Jul 16 22:28:28.802872 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:28.802782    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" event={"ID":"34442e52-c511-4b5d-b9ee-228691dfbc86","Type":"ContainerDied","Data":"e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197"}
Jul 16 22:28:28.802872 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:28.802824    2581 scope.go:117] "RemoveContainer" containerID="75b523b7884e263c0071bf26a1c40efd7ce26252cc62963cc22782b979be3a91"
Jul 16 22:28:28.803049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:28.803031    2581 scope.go:117] "RemoveContainer" containerID="e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197"
Jul 16 22:28:28.803253 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:28.803217    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9_odh-ai-gateway-infra(34442e52-c511-4b5d-b9ee-228691dfbc86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86"
Jul 16 22:28:29.808573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:29.808547    2581 scope.go:117] "RemoveContainer" containerID="e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197"
Jul 16 22:28:29.808953 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:29.808719    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9_odh-ai-gateway-infra(34442e52-c511-4b5d-b9ee-228691dfbc86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86"
Jul 16 22:28:42.155431 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:42.155393    2581 scope.go:117] "RemoveContainer" containerID="eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19"
Jul 16 22:28:42.866794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:42.866768    2581 generic.go:356] "Generic (PLEG): container finished" podID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerID="779c9df953395977936d5912974cf05a06705f007179a7c77dbe8afd2d75d7db" exitCode=6
Jul 16 22:28:42.866901 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:42.866836    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerDied","Data":"779c9df953395977936d5912974cf05a06705f007179a7c77dbe8afd2d75d7db"}
Jul 16 22:28:42.866901 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:42.866896    2581 scope.go:117] "RemoveContainer" containerID="eabd52f6a8e876c781ee28bb8d0c527e0124269763ef53f80a20aaaaba99ee19"
Jul 16 22:28:42.867210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:42.867190    2581 scope.go:117] "RemoveContainer" containerID="779c9df953395977936d5912974cf05a06705f007179a7c77dbe8afd2d75d7db"
Jul 16 22:28:42.867447 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:42.867421    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf_odh-ai-gateway-infra(f3d8d081-7d8a-4585-9bcf-06c0545484cb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb"
Jul 16 22:28:43.894078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:43.894044    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:44.022863 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.022835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:44.137802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.137767    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca\") pod \"f3d8d081-7d8a-4585-9bcf-06c0545484cb\" (UID: \"f3d8d081-7d8a-4585-9bcf-06c0545484cb\") "
Jul 16 22:28:44.138101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.138076    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f3d8d081-7d8a-4585-9bcf-06c0545484cb" (UID: "f3d8d081-7d8a-4585-9bcf-06c0545484cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:44.239071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.238947    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3d8d081-7d8a-4585-9bcf-06c0545484cb-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:28:44.878048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.878020    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"
Jul 16 22:28:44.878259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.878019    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf" event={"ID":"f3d8d081-7d8a-4585-9bcf-06c0545484cb","Type":"ContainerDied","Data":"2b683f10f62f7d746c7bca544b4589139a6cc449faf6f982f86bb50362cfa024"}
Jul 16 22:28:44.878259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.878160    2581 scope.go:117] "RemoveContainer" containerID="779c9df953395977936d5912974cf05a06705f007179a7c77dbe8afd2d75d7db"
Jul 16 22:28:44.900802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.900773    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:44.907399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:44.907377    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-qm4cf"]
Jul 16 22:28:45.155421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.155341    2581 scope.go:117] "RemoveContainer" containerID="e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197"
Jul 16 22:28:45.160671 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.160646    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" path="/var/lib/kubelet/pods/f3d8d081-7d8a-4585-9bcf-06c0545484cb/volumes"
Jul 16 22:28:45.884494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.884463    2581 generic.go:356] "Generic (PLEG): container finished" podID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerID="4f450a260e001fc98d199c1435be731fbc1318b257ee60982427098c09b123dc" exitCode=6
Jul 16 22:28:45.884708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.884543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" event={"ID":"34442e52-c511-4b5d-b9ee-228691dfbc86","Type":"ContainerDied","Data":"4f450a260e001fc98d199c1435be731fbc1318b257ee60982427098c09b123dc"}
Jul 16 22:28:45.884708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.884608    2581 scope.go:117] "RemoveContainer" containerID="e0a74761644e7b255bc45a01e43db64a08e8fd5d22b6b329a53b34f7c7897197"
Jul 16 22:28:45.884959 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:45.884939    2581 scope.go:117] "RemoveContainer" containerID="4f450a260e001fc98d199c1435be731fbc1318b257ee60982427098c09b123dc"
Jul 16 22:28:45.885337 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:45.885312    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9_odh-ai-gateway-infra(34442e52-c511-4b5d-b9ee-228691dfbc86)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86"
Jul 16 22:28:46.911587 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:46.911555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:47.043009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.042982    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:47.164048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.163960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca\") pod \"34442e52-c511-4b5d-b9ee-228691dfbc86\" (UID: \"34442e52-c511-4b5d-b9ee-228691dfbc86\") "
Jul 16 22:28:47.164315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.164293    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "34442e52-c511-4b5d-b9ee-228691dfbc86" (UID: "34442e52-c511-4b5d-b9ee-228691dfbc86"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:47.264626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.264585    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34442e52-c511-4b5d-b9ee-228691dfbc86-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:28:47.897537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.897508    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"
Jul 16 22:28:47.897713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.897509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9" event={"ID":"34442e52-c511-4b5d-b9ee-228691dfbc86","Type":"ContainerDied","Data":"2217d58033bef58853a6673f711aea29f3ef49d1b60065c4af3e5f30bdaf6338"}
Jul 16 22:28:47.897713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.897636    2581 scope.go:117] "RemoveContainer" containerID="4f450a260e001fc98d199c1435be731fbc1318b257ee60982427098c09b123dc"
Jul 16 22:28:47.918139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.918115    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:47.924379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:47.924356    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lvsj9"]
Jul 16 22:28:49.165308 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:49.165268    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" path="/var/lib/kubelet/pods/34442e52-c511-4b5d-b9ee-228691dfbc86/volumes"
Jul 16 22:28:54.280732 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.280692    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281144    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281162    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281173    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281180    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281190    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281197    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281210    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281218    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281252    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281270    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281278    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281409    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281432    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:28:54.281555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.281441    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:28:54.286009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.285988    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:28:54.288890 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.288873    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:54.289040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.289023    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:28:54.292626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.292605    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:28:54.429218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.429180    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq\" (UID: \"dbce26cd-77d8-4af3-982e-c8fcda4edd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:28:54.530108 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.530079    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq\" (UID: \"dbce26cd-77d8-4af3-982e-c8fcda4edd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:28:54.530681 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.530659    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq\" (UID: \"dbce26cd-77d8-4af3-982e-c8fcda4edd78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:28:54.597398 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.597368    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:28:54.732465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.732443    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:28:54.734099 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:28:54.734071    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbce26cd_77d8_4af3_982e_c8fcda4edd78.slice/crio-19ddb9ded5906e8d9258194c54422705e3bd7fe4561099b0409878da32daf176 WatchSource:0}: Error finding container 19ddb9ded5906e8d9258194c54422705e3bd7fe4561099b0409878da32daf176: Status 404 returned error can't find the container with id 19ddb9ded5906e8d9258194c54422705e3bd7fe4561099b0409878da32daf176
Jul 16 22:28:54.930506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:54.930410    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" event={"ID":"dbce26cd-77d8-4af3-982e-c8fcda4edd78","Type":"ContainerStarted","Data":"19ddb9ded5906e8d9258194c54422705e3bd7fe4561099b0409878da32daf176"}
Jul 16 22:28:55.936416 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:55.936379    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerID="826ed0f70a2a38ab77ac66a4e900a6f236de1421eef8fb1790d7a6eee4a233fb" exitCode=6
Jul 16 22:28:55.936799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:55.936429    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" event={"ID":"dbce26cd-77d8-4af3-982e-c8fcda4edd78","Type":"ContainerDied","Data":"826ed0f70a2a38ab77ac66a4e900a6f236de1421eef8fb1790d7a6eee4a233fb"}
Jul 16 22:28:55.936799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:55.936679    2581 scope.go:117] "RemoveContainer" containerID="826ed0f70a2a38ab77ac66a4e900a6f236de1421eef8fb1790d7a6eee4a233fb"
Jul 16 22:28:56.943168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:56.943135    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerID="9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e" exitCode=6
Jul 16 22:28:56.943618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:56.943185    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" event={"ID":"dbce26cd-77d8-4af3-982e-c8fcda4edd78","Type":"ContainerDied","Data":"9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e"}
Jul 16 22:28:56.943618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:56.943252    2581 scope.go:117] "RemoveContainer" containerID="826ed0f70a2a38ab77ac66a4e900a6f236de1421eef8fb1790d7a6eee4a233fb"
Jul 16 22:28:56.943618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:56.943477    2581 scope.go:117] "RemoveContainer" containerID="9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e"
Jul 16 22:28:56.943738 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:56.943697    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq_odh-ai-gateway-infra(dbce26cd-77d8-4af3-982e-c8fcda4edd78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78"
Jul 16 22:28:57.949245 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:28:57.949202    2581 scope.go:117] "RemoveContainer" containerID="9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e"
Jul 16 22:28:57.949624 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:28:57.949408    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq_odh-ai-gateway-infra(dbce26cd-77d8-4af3-982e-c8fcda4edd78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78"
Jul 16 22:29:02.740453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.740419    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:02.740824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.740782    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="34442e52-c511-4b5d-b9ee-228691dfbc86" containerName="revoke-keys"
Jul 16 22:29:02.740824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.740796    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d8d081-7d8a-4585-9bcf-06c0545484cb" containerName="revoke-keys"
Jul 16 22:29:02.743467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.743451    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:02.754059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.754035    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:02.903430 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:02.903390    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-gs79p\" (UID: \"de2e921c-e07c-494f-8da3-4d5c7534e682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:03.004279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.004166    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-gs79p\" (UID: \"de2e921c-e07c-494f-8da3-4d5c7534e682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:03.004922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.004899    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-gs79p\" (UID: \"de2e921c-e07c-494f-8da3-4d5c7534e682\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:03.053291 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.053260    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:03.181351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.181324    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:03.183408 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:29:03.183378    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde2e921c_e07c_494f_8da3_4d5c7534e682.slice/crio-9d78316369002c4b853fd2d9ffb2d8891ca5ff2904903bd586f01f332baf7019 WatchSource:0}: Error finding container 9d78316369002c4b853fd2d9ffb2d8891ca5ff2904903bd586f01f332baf7019: Status 404 returned error can't find the container with id 9d78316369002c4b853fd2d9ffb2d8891ca5ff2904903bd586f01f332baf7019
Jul 16 22:29:03.976778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.976741    2581 generic.go:356] "Generic (PLEG): container finished" podID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerID="d132a9cd2bc6ac3881e8651edb7e7b3d18868370ec4c9cf227afa6faa670799c" exitCode=7
Jul 16 22:29:03.977170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.976828    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" event={"ID":"de2e921c-e07c-494f-8da3-4d5c7534e682","Type":"ContainerDied","Data":"d132a9cd2bc6ac3881e8651edb7e7b3d18868370ec4c9cf227afa6faa670799c"}
Jul 16 22:29:03.977170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.976887    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" event={"ID":"de2e921c-e07c-494f-8da3-4d5c7534e682","Type":"ContainerStarted","Data":"9d78316369002c4b853fd2d9ffb2d8891ca5ff2904903bd586f01f332baf7019"}
Jul 16 22:29:03.977170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:03.976972    2581 scope.go:117] "RemoveContainer" containerID="d132a9cd2bc6ac3881e8651edb7e7b3d18868370ec4c9cf227afa6faa670799c"
Jul 16 22:29:04.983538 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:04.983503    2581 generic.go:356] "Generic (PLEG): container finished" podID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerID="f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069" exitCode=7
Jul 16 22:29:04.983999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:04.983620    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" event={"ID":"de2e921c-e07c-494f-8da3-4d5c7534e682","Type":"ContainerDied","Data":"f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069"}
Jul 16 22:29:04.983999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:04.983674    2581 scope.go:117] "RemoveContainer" containerID="d132a9cd2bc6ac3881e8651edb7e7b3d18868370ec4c9cf227afa6faa670799c"
Jul 16 22:29:04.983999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:04.983951    2581 scope.go:117] "RemoveContainer" containerID="f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069"
Jul 16 22:29:04.984154 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:04.984136    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-gs79p_odh-ai-gateway-infra(de2e921c-e07c-494f-8da3-4d5c7534e682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682"
Jul 16 22:29:05.989992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:05.989964    2581 scope.go:117] "RemoveContainer" containerID="f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069"
Jul 16 22:29:05.990367 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:05.990160    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-gs79p_odh-ai-gateway-infra(de2e921c-e07c-494f-8da3-4d5c7534e682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682"
Jul 16 22:29:12.155444 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:12.155412    2581 scope.go:117] "RemoveContainer" containerID="9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e"
Jul 16 22:29:13.029206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:13.029115    2581 generic.go:356] "Generic (PLEG): container finished" podID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerID="ff8528d38faea4826514082bb78fd9dc27cdfe05753df5e36a929a3319e1169c" exitCode=6
Jul 16 22:29:13.029206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:13.029180    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" event={"ID":"dbce26cd-77d8-4af3-982e-c8fcda4edd78","Type":"ContainerDied","Data":"ff8528d38faea4826514082bb78fd9dc27cdfe05753df5e36a929a3319e1169c"}
Jul 16 22:29:13.029451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:13.029248    2581 scope.go:117] "RemoveContainer" containerID="9f969b845b1aede59925125b8c30e9342bbd9774a60daaaf0b8ac1508d3e558e"
Jul 16 22:29:13.029534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:13.029517    2581 scope.go:117] "RemoveContainer" containerID="ff8528d38faea4826514082bb78fd9dc27cdfe05753df5e36a929a3319e1169c"
Jul 16 22:29:13.029748 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:13.029730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq_odh-ai-gateway-infra(dbce26cd-77d8-4af3-982e-c8fcda4edd78)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78"
Jul 16 22:29:14.057383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:14.057348    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:29:14.193739 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:14.193708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:29:14.297373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:14.297334    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca\") pod \"dbce26cd-77d8-4af3-982e-c8fcda4edd78\" (UID: \"dbce26cd-77d8-4af3-982e-c8fcda4edd78\") "
Jul 16 22:29:14.297709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:14.297685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dbce26cd-77d8-4af3-982e-c8fcda4edd78" (UID: "dbce26cd-77d8-4af3-982e-c8fcda4edd78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:14.398094 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:14.398058    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dbce26cd-77d8-4af3-982e-c8fcda4edd78-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:29:15.040267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.040222    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"
Jul 16 22:29:15.040267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.040261    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq" event={"ID":"dbce26cd-77d8-4af3-982e-c8fcda4edd78","Type":"ContainerDied","Data":"19ddb9ded5906e8d9258194c54422705e3bd7fe4561099b0409878da32daf176"}
Jul 16 22:29:15.040496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.040320    2581 scope.go:117] "RemoveContainer" containerID="ff8528d38faea4826514082bb78fd9dc27cdfe05753df5e36a929a3319e1169c"
Jul 16 22:29:15.061568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.061542    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:29:15.066942 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.066920    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xcxlq"]
Jul 16 22:29:15.125489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125453    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:15.125780 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125767    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125780    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125797    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125803    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125858    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125865    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.125928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.125871    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:15.130012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.129990    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:15.138641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.138589    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:15.160698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.160658    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" path="/var/lib/kubelet/pods/dbce26cd-77d8-4af3-982e-c8fcda4edd78/volumes"
Jul 16 22:29:15.205488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.205445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7\" (UID: \"8adb2289-83c2-475f-bb09-4a2ecbe970d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:15.306530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.306437    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7\" (UID: \"8adb2289-83c2-475f-bb09-4a2ecbe970d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:15.307045 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.307024    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7\" (UID: \"8adb2289-83c2-475f-bb09-4a2ecbe970d2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:15.440666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.440634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:15.569433 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:15.569347    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:15.572854 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:29:15.572825    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8adb2289_83c2_475f_bb09_4a2ecbe970d2.slice/crio-c835da203799d1be082f9515277b9f35b4a091cebbd6debb853eaa1f54c80141 WatchSource:0}: Error finding container c835da203799d1be082f9515277b9f35b4a091cebbd6debb853eaa1f54c80141: Status 404 returned error can't find the container with id c835da203799d1be082f9515277b9f35b4a091cebbd6debb853eaa1f54c80141
Jul 16 22:29:16.046531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:16.046494    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerStarted","Data":"bc757cc2070be7df2b75f5ee1da06da9655038b04fe7eefa85f5c877492219ea"}
Jul 16 22:29:16.046742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:16.046551    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerStarted","Data":"c835da203799d1be082f9515277b9f35b4a091cebbd6debb853eaa1f54c80141"}
Jul 16 22:29:17.053915 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:17.053877    2581 generic.go:356] "Generic (PLEG): container finished" podID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerID="bc757cc2070be7df2b75f5ee1da06da9655038b04fe7eefa85f5c877492219ea" exitCode=6
Jul 16 22:29:17.054408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:17.053933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerDied","Data":"bc757cc2070be7df2b75f5ee1da06da9655038b04fe7eefa85f5c877492219ea"}
Jul 16 22:29:17.054408 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:17.054176    2581 scope.go:117] "RemoveContainer" containerID="bc757cc2070be7df2b75f5ee1da06da9655038b04fe7eefa85f5c877492219ea"
Jul 16 22:29:17.155524 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:17.155484    2581 scope.go:117] "RemoveContainer" containerID="f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069"
Jul 16 22:29:18.060540 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.060451    2581 generic.go:356] "Generic (PLEG): container finished" podID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerID="39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918" exitCode=6
Jul 16 22:29:18.060994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.060531    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerDied","Data":"39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918"}
Jul 16 22:29:18.060994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.060605    2581 scope.go:117] "RemoveContainer" containerID="bc757cc2070be7df2b75f5ee1da06da9655038b04fe7eefa85f5c877492219ea"
Jul 16 22:29:18.060994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.060719    2581 scope.go:117] "RemoveContainer" containerID="39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918"
Jul 16 22:29:18.060994 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:18.060971    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7_odh-ai-gateway-infra(8adb2289-83c2-475f-bb09-4a2ecbe970d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2"
Jul 16 22:29:18.062446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.062428    2581 generic.go:356] "Generic (PLEG): container finished" podID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerID="63519244cfa09357d4c420f1301575c2b10a5078accdfb999abb840a1e0e5963" exitCode=7
Jul 16 22:29:18.062507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.062470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" event={"ID":"de2e921c-e07c-494f-8da3-4d5c7534e682","Type":"ContainerDied","Data":"63519244cfa09357d4c420f1301575c2b10a5078accdfb999abb840a1e0e5963"}
Jul 16 22:29:18.062725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.062712    2581 scope.go:117] "RemoveContainer" containerID="63519244cfa09357d4c420f1301575c2b10a5078accdfb999abb840a1e0e5963"
Jul 16 22:29:18.062873 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:18.062859    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-gs79p_odh-ai-gateway-infra(de2e921c-e07c-494f-8da3-4d5c7534e682)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682"
Jul 16 22:29:18.071187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:18.071165    2581 scope.go:117] "RemoveContainer" containerID="f361aceb8c3a16436c2a67585b739e190e587b49802d9b01492cb55bb9e41069"
Jul 16 22:29:19.069582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.069556    2581 scope.go:117] "RemoveContainer" containerID="39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918"
Jul 16 22:29:19.070048 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:19.069809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7_odh-ai-gateway-infra(8adb2289-83c2-475f-bb09-4a2ecbe970d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2"
Jul 16 22:29:19.108591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.108555    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:19.253616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.253586    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:19.340531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.340499    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca\") pod \"de2e921c-e07c-494f-8da3-4d5c7534e682\" (UID: \"de2e921c-e07c-494f-8da3-4d5c7534e682\") "
Jul 16 22:29:19.340883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.340861    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "de2e921c-e07c-494f-8da3-4d5c7534e682" (UID: "de2e921c-e07c-494f-8da3-4d5c7534e682"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:19.442093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:19.442047    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/de2e921c-e07c-494f-8da3-4d5c7534e682-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:29:20.077507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:20.077468    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p" event={"ID":"de2e921c-e07c-494f-8da3-4d5c7534e682","Type":"ContainerDied","Data":"9d78316369002c4b853fd2d9ffb2d8891ca5ff2904903bd586f01f332baf7019"}
Jul 16 22:29:20.078009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:20.077511    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"
Jul 16 22:29:20.078009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:20.077533    2581 scope.go:117] "RemoveContainer" containerID="63519244cfa09357d4c420f1301575c2b10a5078accdfb999abb840a1e0e5963"
Jul 16 22:29:20.099663 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:20.099626    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:20.104605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:20.104575    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-gs79p"]
Jul 16 22:29:21.160397 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:21.160361    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" path="/var/lib/kubelet/pods/de2e921c-e07c-494f-8da3-4d5c7534e682/volumes"
Jul 16 22:29:26.687995 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.687958    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688325    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688340    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688355    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688360    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688377    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbce26cd-77d8-4af3-982e-c8fcda4edd78" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688434    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.688477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.688442    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:26.692747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.692729    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:26.700892 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.700854    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:26.806483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.806445    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5\" (UID: \"6dbb66e0-a7df-4062-b89f-df23328ca068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:26.906983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.906953    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5\" (UID: \"6dbb66e0-a7df-4062-b89f-df23328ca068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:26.907616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:26.907596    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5\" (UID: \"6dbb66e0-a7df-4062-b89f-df23328ca068\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:27.004147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:27.004071    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:27.150418 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:29:27.150379    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dbb66e0_a7df_4062_b89f_df23328ca068.slice/crio-676d6f9e0e72dd46bb39d9752dafdd8866d51c0fa8738cc91e9a52bf357696db WatchSource:0}: Error finding container 676d6f9e0e72dd46bb39d9752dafdd8866d51c0fa8738cc91e9a52bf357696db: Status 404 returned error can't find the container with id 676d6f9e0e72dd46bb39d9752dafdd8866d51c0fa8738cc91e9a52bf357696db
Jul 16 22:29:27.150963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:27.150905    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:28.117480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:28.117389    2581 generic.go:356] "Generic (PLEG): container finished" podID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerID="473d34420f9a70a90ea2f04459bdfac6537c8c7a7999b8824b836245d8479402" exitCode=6
Jul 16 22:29:28.117480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:28.117436    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" event={"ID":"6dbb66e0-a7df-4062-b89f-df23328ca068","Type":"ContainerDied","Data":"473d34420f9a70a90ea2f04459bdfac6537c8c7a7999b8824b836245d8479402"}
Jul 16 22:29:28.117894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:28.117484    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" event={"ID":"6dbb66e0-a7df-4062-b89f-df23328ca068","Type":"ContainerStarted","Data":"676d6f9e0e72dd46bb39d9752dafdd8866d51c0fa8738cc91e9a52bf357696db"}
Jul 16 22:29:28.117894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:28.117697    2581 scope.go:117] "RemoveContainer" containerID="473d34420f9a70a90ea2f04459bdfac6537c8c7a7999b8824b836245d8479402"
Jul 16 22:29:29.123607 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:29.123575    2581 generic.go:356] "Generic (PLEG): container finished" podID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerID="aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72" exitCode=6
Jul 16 22:29:29.124076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:29.123663    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" event={"ID":"6dbb66e0-a7df-4062-b89f-df23328ca068","Type":"ContainerDied","Data":"aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72"}
Jul 16 22:29:29.124076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:29.123733    2581 scope.go:117] "RemoveContainer" containerID="473d34420f9a70a90ea2f04459bdfac6537c8c7a7999b8824b836245d8479402"
Jul 16 22:29:29.124076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:29.123819    2581 scope.go:117] "RemoveContainer" containerID="aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72"
Jul 16 22:29:29.124076 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:29.124051    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5_odh-ai-gateway-infra(6dbb66e0-a7df-4062-b89f-df23328ca068)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068"
Jul 16 22:29:30.129820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:30.129790    2581 scope.go:117] "RemoveContainer" containerID="aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72"
Jul 16 22:29:30.130191 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:30.129978    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5_odh-ai-gateway-infra(6dbb66e0-a7df-4062-b89f-df23328ca068)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068"
Jul 16 22:29:31.155735 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:31.155703    2581 scope.go:117] "RemoveContainer" containerID="39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918"
Jul 16 22:29:32.139727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:32.139686    2581 generic.go:356] "Generic (PLEG): container finished" podID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerID="a4028829b5c1d7f03b65b54889951f0f9e7edd3c98f5da3726aa9ea96ea33fd2" exitCode=6
Jul 16 22:29:32.139898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:32.139731    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerDied","Data":"a4028829b5c1d7f03b65b54889951f0f9e7edd3c98f5da3726aa9ea96ea33fd2"}
Jul 16 22:29:32.139898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:32.139800    2581 scope.go:117] "RemoveContainer" containerID="39c323bf6971d41c60afc03a4d2a55cf282f08ea1438432837c3689c185bb918"
Jul 16 22:29:32.140120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:32.140097    2581 scope.go:117] "RemoveContainer" containerID="a4028829b5c1d7f03b65b54889951f0f9e7edd3c98f5da3726aa9ea96ea33fd2"
Jul 16 22:29:32.140381 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:32.140358    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7_odh-ai-gateway-infra(8adb2289-83c2-475f-bb09-4a2ecbe970d2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2"
Jul 16 22:29:33.203118 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:33.203084    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:33.333202 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:33.333175    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:33.463441 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:33.463340    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca\") pod \"8adb2289-83c2-475f-bb09-4a2ecbe970d2\" (UID: \"8adb2289-83c2-475f-bb09-4a2ecbe970d2\") "
Jul 16 22:29:33.463708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:33.463684    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8adb2289-83c2-475f-bb09-4a2ecbe970d2" (UID: "8adb2289-83c2-475f-bb09-4a2ecbe970d2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:33.564077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:33.564032    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8adb2289-83c2-475f-bb09-4a2ecbe970d2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:29:34.150984 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:34.150954    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"
Jul 16 22:29:34.151158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:34.150994    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7" event={"ID":"8adb2289-83c2-475f-bb09-4a2ecbe970d2","Type":"ContainerDied","Data":"c835da203799d1be082f9515277b9f35b4a091cebbd6debb853eaa1f54c80141"}
Jul 16 22:29:34.151158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:34.151055    2581 scope.go:117] "RemoveContainer" containerID="a4028829b5c1d7f03b65b54889951f0f9e7edd3c98f5da3726aa9ea96ea33fd2"
Jul 16 22:29:34.175356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:34.175310    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:34.183703 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:34.183677    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-q6vn7"]
Jul 16 22:29:35.160089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:35.160056    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" path="/var/lib/kubelet/pods/8adb2289-83c2-475f-bb09-4a2ecbe970d2/volumes"
Jul 16 22:29:43.160123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:43.160085    2581 scope.go:117] "RemoveContainer" containerID="aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72"
Jul 16 22:29:44.195866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:44.195778    2581 generic.go:356] "Generic (PLEG): container finished" podID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerID="e8fa529a7e67737314dda8def84fe6758fec894d3f37d1b81d3149e75918e017" exitCode=6
Jul 16 22:29:44.195866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:44.195848    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" event={"ID":"6dbb66e0-a7df-4062-b89f-df23328ca068","Type":"ContainerDied","Data":"e8fa529a7e67737314dda8def84fe6758fec894d3f37d1b81d3149e75918e017"}
Jul 16 22:29:44.196321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:44.195910    2581 scope.go:117] "RemoveContainer" containerID="aeb3d45c3a0f5d095e71b0565d4942cbb8cd47f752ef3645a2593d8d3d6ddc72"
Jul 16 22:29:44.196321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:44.196273    2581 scope.go:117] "RemoveContainer" containerID="e8fa529a7e67737314dda8def84fe6758fec894d3f37d1b81d3149e75918e017"
Jul 16 22:29:44.196502 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:44.196482    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5_odh-ai-gateway-infra(6dbb66e0-a7df-4062-b89f-df23328ca068)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068"
Jul 16 22:29:45.248305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:45.248271    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:45.379000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:45.378977    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:45.469432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:45.469405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca\") pod \"6dbb66e0-a7df-4062-b89f-df23328ca068\" (UID: \"6dbb66e0-a7df-4062-b89f-df23328ca068\") "
Jul 16 22:29:45.469744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:45.469722    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6dbb66e0-a7df-4062-b89f-df23328ca068" (UID: "6dbb66e0-a7df-4062-b89f-df23328ca068"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:45.570441 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:45.570371    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6dbb66e0-a7df-4062-b89f-df23328ca068-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:29:46.207387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:46.207354    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"
Jul 16 22:29:46.207387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:46.207377    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5" event={"ID":"6dbb66e0-a7df-4062-b89f-df23328ca068","Type":"ContainerDied","Data":"676d6f9e0e72dd46bb39d9752dafdd8866d51c0fa8738cc91e9a52bf357696db"}
Jul 16 22:29:46.207653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:46.207431    2581 scope.go:117] "RemoveContainer" containerID="e8fa529a7e67737314dda8def84fe6758fec894d3f37d1b81d3149e75918e017"
Jul 16 22:29:46.230653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:46.230623    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:46.236723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:46.236701    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-qw8f5"]
Jul 16 22:29:47.160339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:47.160304    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" path="/var/lib/kubelet/pods/6dbb66e0-a7df-4062-b89f-df23328ca068/volumes"
Jul 16 22:29:52.824691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824657    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824957    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824968    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824981    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824986    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.824997    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825003    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825009    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825014    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825019    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825024    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825033    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825037    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825084    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825092    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="de2e921c-e07c-494f-8da3-4d5c7534e682" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825099    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825104    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825109    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.825162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.825115    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dbb66e0-a7df-4062-b89f-df23328ca068" containerName="revoke-keys"
Jul 16 22:29:52.829414 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.829395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:29:52.831739 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.831715    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qvh26\" (UID: \"add2fd15-03a4-4c63-ad18-a9a3eeb082f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:29:52.833082 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.833064    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:29:52.833946 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.833931    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:29:52.847650 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.847629    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:29:52.932537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.932504    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qvh26\" (UID: \"add2fd15-03a4-4c63-ad18-a9a3eeb082f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:29:52.933045 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:52.933027    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qvh26\" (UID: \"add2fd15-03a4-4c63-ad18-a9a3eeb082f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:29:53.142314 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:53.142290    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:29:53.149509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:53.149487    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:29:53.289027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:53.289003    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:29:53.291518 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:29:53.291488    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadd2fd15_03a4_4c63_ad18_a9a3eeb082f2.slice/crio-e72c3e9bfe11d9397a29547f0bbbee25e8d3e9dbb3860471c20d7ce60ff0d933 WatchSource:0}: Error finding container e72c3e9bfe11d9397a29547f0bbbee25e8d3e9dbb3860471c20d7ce60ff0d933: Status 404 returned error can't find the container with id e72c3e9bfe11d9397a29547f0bbbee25e8d3e9dbb3860471c20d7ce60ff0d933
Jul 16 22:29:54.246028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.245943    2581 generic.go:356] "Generic (PLEG): container finished" podID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerID="f2e35282ddaab5975821a470295afce186fbc868f5e229320dbdcc2b5d1a6681" exitCode=7
Jul 16 22:29:54.246028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.245988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" event={"ID":"add2fd15-03a4-4c63-ad18-a9a3eeb082f2","Type":"ContainerDied","Data":"f2e35282ddaab5975821a470295afce186fbc868f5e229320dbdcc2b5d1a6681"}
Jul 16 22:29:54.246500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.246052    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" event={"ID":"add2fd15-03a4-4c63-ad18-a9a3eeb082f2","Type":"ContainerStarted","Data":"e72c3e9bfe11d9397a29547f0bbbee25e8d3e9dbb3860471c20d7ce60ff0d933"}
Jul 16 22:29:54.246500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.246217    2581 scope.go:117] "RemoveContainer" containerID="f2e35282ddaab5975821a470295afce186fbc868f5e229320dbdcc2b5d1a6681"
Jul 16 22:29:54.355336 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.355309    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:29:54.355638 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.355626    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:54.355681 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.355640    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:54.355715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.355703    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8adb2289-83c2-475f-bb09-4a2ecbe970d2" containerName="revoke-keys"
Jul 16 22:29:54.359761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.359746    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:29:54.370116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.370077    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:29:54.443984 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.443951    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt\" (UID: \"7ffaf08d-9fee-4cb7-a3b0-ccb039603364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:29:54.544945 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.544878    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt\" (UID: \"7ffaf08d-9fee-4cb7-a3b0-ccb039603364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:29:54.545536 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.545514    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt\" (UID: \"7ffaf08d-9fee-4cb7-a3b0-ccb039603364\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:29:54.669686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.669654    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:29:54.825053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:54.825027    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:29:54.826655 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:29:54.826629    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffaf08d_9fee_4cb7_a3b0_ccb039603364.slice/crio-8434c63c37a630dd5f1f69ac9c641f5e532cc0ccab359cef8c19927201844c5c WatchSource:0}: Error finding container 8434c63c37a630dd5f1f69ac9c641f5e532cc0ccab359cef8c19927201844c5c: Status 404 returned error can't find the container with id 8434c63c37a630dd5f1f69ac9c641f5e532cc0ccab359cef8c19927201844c5c
Jul 16 22:29:55.252152 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.252109    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerStarted","Data":"eab053d187c3eb24dc8d07ba86b933ceef5f30ea2345a486ea17cd35fb97c031"}
Jul 16 22:29:55.252629 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.252176    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerStarted","Data":"8434c63c37a630dd5f1f69ac9c641f5e532cc0ccab359cef8c19927201844c5c"}
Jul 16 22:29:55.253997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.253971    2581 generic.go:356] "Generic (PLEG): container finished" podID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerID="9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206" exitCode=7
Jul 16 22:29:55.254116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.254033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" event={"ID":"add2fd15-03a4-4c63-ad18-a9a3eeb082f2","Type":"ContainerDied","Data":"9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206"}
Jul 16 22:29:55.254116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.254086    2581 scope.go:117] "RemoveContainer" containerID="f2e35282ddaab5975821a470295afce186fbc868f5e229320dbdcc2b5d1a6681"
Jul 16 22:29:55.254278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.254205    2581 scope.go:117] "RemoveContainer" containerID="9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206"
Jul 16 22:29:55.254409 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:55.254387    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qvh26_odh-ai-gateway-infra(add2fd15-03a4-4c63-ad18-a9a3eeb082f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2"
Jul 16 22:29:55.290827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:55.290781    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" podStartSLOduration=1.290767388 podStartE2EDuration="1.290767388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:29:55.290625884 +0000 UTC m=+2582.620065786" watchObservedRunningTime="2026-07-16 22:29:55.290767388 +0000 UTC m=+2582.620207290"
Jul 16 22:29:56.260793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:56.260758    2581 scope.go:117] "RemoveContainer" containerID="9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206"
Jul 16 22:29:56.261212 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:56.260988    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qvh26_odh-ai-gateway-infra(add2fd15-03a4-4c63-ad18-a9a3eeb082f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2"
Jul 16 22:29:56.262051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:56.262030    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerID="eab053d187c3eb24dc8d07ba86b933ceef5f30ea2345a486ea17cd35fb97c031" exitCode=6
Jul 16 22:29:56.262125 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:56.262065    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerDied","Data":"eab053d187c3eb24dc8d07ba86b933ceef5f30ea2345a486ea17cd35fb97c031"}
Jul 16 22:29:56.262271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:56.262259    2581 scope.go:117] "RemoveContainer" containerID="eab053d187c3eb24dc8d07ba86b933ceef5f30ea2345a486ea17cd35fb97c031"
Jul 16 22:29:57.268599 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:57.268503    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerID="da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13" exitCode=6
Jul 16 22:29:57.268599 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:57.268562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerDied","Data":"da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13"}
Jul 16 22:29:57.269113 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:57.268612    2581 scope.go:117] "RemoveContainer" containerID="eab053d187c3eb24dc8d07ba86b933ceef5f30ea2345a486ea17cd35fb97c031"
Jul 16 22:29:57.269113 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:57.268831    2581 scope.go:117] "RemoveContainer" containerID="da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13"
Jul 16 22:29:57.269113 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:57.269062    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt_odh-ai-gateway-infra(7ffaf08d-9fee-4cb7-a3b0-ccb039603364)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364"
Jul 16 22:29:58.274619 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:29:58.274596    2581 scope.go:117] "RemoveContainer" containerID="da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13"
Jul 16 22:29:58.274986 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:29:58.274769    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt_odh-ai-gateway-infra(7ffaf08d-9fee-4cb7-a3b0-ccb039603364)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364"
Jul 16 22:30:00.155594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:00.155560    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:00.158679 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:00.158661    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"
Jul 16 22:30:00.167189 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:00.167170    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"
Jul 16 22:30:00.185337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:00.185310    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:00.317499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:00.317467    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:00.319839 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:30:00.319811    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9bce9913_1cdd_4c54_b7da_a526d0c6ea42.slice/crio-cbf3a105f4251920095d4997ca737ba08aafb04ccc2c4ec50e6b819ef05eda8d WatchSource:0}: Error finding container cbf3a105f4251920095d4997ca737ba08aafb04ccc2c4ec50e6b819ef05eda8d: Status 404 returned error can't find the container with id cbf3a105f4251920095d4997ca737ba08aafb04ccc2c4ec50e6b819ef05eda8d
Jul 16 22:30:01.289429 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:01.289393    2581 generic.go:356] "Generic (PLEG): container finished" podID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerID="ab8d8f81e9178c736cb9a8d24aea5a7b7a60f3214e576bf5e487bbddc70ac0fd" exitCode=7
Jul 16 22:30:01.289782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:01.289466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" event={"ID":"9bce9913-1cdd-4c54-b7da-a526d0c6ea42","Type":"ContainerDied","Data":"ab8d8f81e9178c736cb9a8d24aea5a7b7a60f3214e576bf5e487bbddc70ac0fd"}
Jul 16 22:30:01.289782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:01.289509    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" event={"ID":"9bce9913-1cdd-4c54-b7da-a526d0c6ea42","Type":"ContainerStarted","Data":"cbf3a105f4251920095d4997ca737ba08aafb04ccc2c4ec50e6b819ef05eda8d"}
Jul 16 22:30:01.289782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:01.289576    2581 scope.go:117] "RemoveContainer" containerID="ab8d8f81e9178c736cb9a8d24aea5a7b7a60f3214e576bf5e487bbddc70ac0fd"
Jul 16 22:30:02.295432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:02.295337    2581 generic.go:356] "Generic (PLEG): container finished" podID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerID="8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287" exitCode=7
Jul 16 22:30:02.295850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:02.295435    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" event={"ID":"9bce9913-1cdd-4c54-b7da-a526d0c6ea42","Type":"ContainerDied","Data":"8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287"}
Jul 16 22:30:02.295850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:02.295503    2581 scope.go:117] "RemoveContainer" containerID="ab8d8f81e9178c736cb9a8d24aea5a7b7a60f3214e576bf5e487bbddc70ac0fd"
Jul 16 22:30:02.295850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:02.295627    2581 scope.go:117] "RemoveContainer" containerID="8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287"
Jul 16 22:30:02.295850 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:02.295840    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp_odh-ai-gateway-infra(9bce9913-1cdd-4c54-b7da-a526d0c6ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42"
Jul 16 22:30:03.302284 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:03.302251    2581 scope.go:117] "RemoveContainer" containerID="8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287"
Jul 16 22:30:03.302647 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:03.302445    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp_odh-ai-gateway-infra(9bce9913-1cdd-4c54-b7da-a526d0c6ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42"
Jul 16 22:30:05.200634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.200601    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:05.205063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.205045    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:05.212470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.212354    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:05.234492 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.234465    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj\" (UID: \"60a72c95-49b7-4c31-8668-82e8ecb889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:05.335633 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.335605    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj\" (UID: \"60a72c95-49b7-4c31-8668-82e8ecb889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:05.336188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.336168    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj\" (UID: \"60a72c95-49b7-4c31-8668-82e8ecb889e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:05.516494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.516415    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:05.647313 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:05.647290    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:05.649428 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:30:05.649398    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60a72c95_49b7_4c31_8668_82e8ecb889e8.slice/crio-801f47cb16cf36d2685faf9af71d5cb5ebda14071edccd0c34c8a59e7a250b14 WatchSource:0}: Error finding container 801f47cb16cf36d2685faf9af71d5cb5ebda14071edccd0c34c8a59e7a250b14: Status 404 returned error can't find the container with id 801f47cb16cf36d2685faf9af71d5cb5ebda14071edccd0c34c8a59e7a250b14
Jul 16 22:30:06.316073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:06.316041    2581 generic.go:356] "Generic (PLEG): container finished" podID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerID="29949b608724d84023fb21b5fc2ad9d35a06f6a107e4caeaafec349f1afad998" exitCode=6
Jul 16 22:30:06.316446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:06.316106    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerDied","Data":"29949b608724d84023fb21b5fc2ad9d35a06f6a107e4caeaafec349f1afad998"}
Jul 16 22:30:06.316446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:06.316158    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerStarted","Data":"801f47cb16cf36d2685faf9af71d5cb5ebda14071edccd0c34c8a59e7a250b14"}
Jul 16 22:30:06.316446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:06.316430    2581 scope.go:117] "RemoveContainer" containerID="29949b608724d84023fb21b5fc2ad9d35a06f6a107e4caeaafec349f1afad998"
Jul 16 22:30:07.323002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:07.322924    2581 generic.go:356] "Generic (PLEG): container finished" podID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerID="3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162" exitCode=6
Jul 16 22:30:07.323437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:07.323015    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerDied","Data":"3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162"}
Jul 16 22:30:07.323437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:07.323086    2581 scope.go:117] "RemoveContainer" containerID="29949b608724d84023fb21b5fc2ad9d35a06f6a107e4caeaafec349f1afad998"
Jul 16 22:30:07.323437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:07.323224    2581 scope.go:117] "RemoveContainer" containerID="3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162"
Jul 16 22:30:07.323566 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:07.323487    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj_odh-ai-gateway-infra(60a72c95-49b7-4c31-8668-82e8ecb889e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8"
Jul 16 22:30:08.155372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:08.155332    2581 scope.go:117] "RemoveContainer" containerID="9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206"
Jul 16 22:30:08.328733 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:08.328710    2581 scope.go:117] "RemoveContainer" containerID="3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162"
Jul 16 22:30:08.329089 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:08.328872    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj_odh-ai-gateway-infra(60a72c95-49b7-4c31-8668-82e8ecb889e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8"
Jul 16 22:30:09.334886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:09.334851    2581 generic.go:356] "Generic (PLEG): container finished" podID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerID="9338a6d17331cf038c802d739851f00368a73300974883107a78c86243c5207a" exitCode=7
Jul 16 22:30:09.335367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:09.334913    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" event={"ID":"add2fd15-03a4-4c63-ad18-a9a3eeb082f2","Type":"ContainerDied","Data":"9338a6d17331cf038c802d739851f00368a73300974883107a78c86243c5207a"}
Jul 16 22:30:09.335367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:09.334958    2581 scope.go:117] "RemoveContainer" containerID="9b5b60feb0b38763772b6226ae6fe97dc6cb789332904c2dd8ba0b6f53ad1206"
Jul 16 22:30:09.335367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:09.335307    2581 scope.go:117] "RemoveContainer" containerID="9338a6d17331cf038c802d739851f00368a73300974883107a78c86243c5207a"
Jul 16 22:30:09.335570 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:09.335547    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qvh26_odh-ai-gateway-infra(add2fd15-03a4-4c63-ad18-a9a3eeb082f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2"
Jul 16 22:30:10.365489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:10.365455    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:30:10.497772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:10.497751    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:30:10.577121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:10.577090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca\") pod \"add2fd15-03a4-4c63-ad18-a9a3eeb082f2\" (UID: \"add2fd15-03a4-4c63-ad18-a9a3eeb082f2\") "
Jul 16 22:30:10.577427 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:10.577406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "add2fd15-03a4-4c63-ad18-a9a3eeb082f2" (UID: "add2fd15-03a4-4c63-ad18-a9a3eeb082f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:10.678629 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:10.678550    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/add2fd15-03a4-4c63-ad18-a9a3eeb082f2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:30:11.346523 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:11.346487    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26" event={"ID":"add2fd15-03a4-4c63-ad18-a9a3eeb082f2","Type":"ContainerDied","Data":"e72c3e9bfe11d9397a29547f0bbbee25e8d3e9dbb3860471c20d7ce60ff0d933"}
Jul 16 22:30:11.346706 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:11.346524    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"
Jul 16 22:30:11.346706 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:11.346558    2581 scope.go:117] "RemoveContainer" containerID="9338a6d17331cf038c802d739851f00368a73300974883107a78c86243c5207a"
Jul 16 22:30:11.367969 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:11.367946    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:30:11.371062 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:11.371044    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qvh26"]
Jul 16 22:30:13.157360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:13.157331    2581 scope.go:117] "RemoveContainer" containerID="da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13"
Jul 16 22:30:13.161266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:13.161218    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" path="/var/lib/kubelet/pods/add2fd15-03a4-4c63-ad18-a9a3eeb082f2/volumes"
Jul 16 22:30:14.361401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:14.361325    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerID="ebc7c6e3d19abc0d922223891fae151f447540efa39b321b7655bf674517e1da" exitCode=6
Jul 16 22:30:14.361803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:14.361396    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerDied","Data":"ebc7c6e3d19abc0d922223891fae151f447540efa39b321b7655bf674517e1da"}
Jul 16 22:30:14.361803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:14.361462    2581 scope.go:117] "RemoveContainer" containerID="da0eb90642b3e0bd4304269b4685800c6428822b98908c9a27d8d643d4198b13"
Jul 16 22:30:14.361803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:14.361744    2581 scope.go:117] "RemoveContainer" containerID="ebc7c6e3d19abc0d922223891fae151f447540efa39b321b7655bf674517e1da"
Jul 16 22:30:14.361951 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:14.361928    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt_odh-ai-gateway-infra(7ffaf08d-9fee-4cb7-a3b0-ccb039603364)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364"
Jul 16 22:30:15.154600 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.154572    2581 scope.go:117] "RemoveContainer" containerID="8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287"
Jul 16 22:30:15.410539 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.410466    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:30:15.542666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.542639    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:30:15.618328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.618295    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca\") pod \"7ffaf08d-9fee-4cb7-a3b0-ccb039603364\" (UID: \"7ffaf08d-9fee-4cb7-a3b0-ccb039603364\") "
Jul 16 22:30:15.618626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.618605    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ffaf08d-9fee-4cb7-a3b0-ccb039603364" (UID: "7ffaf08d-9fee-4cb7-a3b0-ccb039603364"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:15.718863 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:15.718793    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffaf08d-9fee-4cb7-a3b0-ccb039603364-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:30:16.372798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.372762    2581 generic.go:356] "Generic (PLEG): container finished" podID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerID="a754d64b9825217e5f79a295e7455763cd15af1183eaa99f4fbffe8e1ed58ee8" exitCode=7
Jul 16 22:30:16.372997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.372833    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" event={"ID":"9bce9913-1cdd-4c54-b7da-a526d0c6ea42","Type":"ContainerDied","Data":"a754d64b9825217e5f79a295e7455763cd15af1183eaa99f4fbffe8e1ed58ee8"}
Jul 16 22:30:16.372997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.372889    2581 scope.go:117] "RemoveContainer" containerID="8433267eb6c368a3fac4ff9bb0ed16a5601b3772807e469464d33f227cc1f287"
Jul 16 22:30:16.373144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.373128    2581 scope.go:117] "RemoveContainer" containerID="a754d64b9825217e5f79a295e7455763cd15af1183eaa99f4fbffe8e1ed58ee8"
Jul 16 22:30:16.373403 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:16.373372    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp_odh-ai-gateway-infra(9bce9913-1cdd-4c54-b7da-a526d0c6ea42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42"
Jul 16 22:30:16.374263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.374224    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"
Jul 16 22:30:16.374382 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.374243    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt" event={"ID":"7ffaf08d-9fee-4cb7-a3b0-ccb039603364","Type":"ContainerDied","Data":"8434c63c37a630dd5f1f69ac9c641f5e532cc0ccab359cef8c19927201844c5c"}
Jul 16 22:30:16.383819 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.383802    2581 scope.go:117] "RemoveContainer" containerID="ebc7c6e3d19abc0d922223891fae151f447540efa39b321b7655bf674517e1da"
Jul 16 22:30:16.449698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.449672    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:30:16.464382 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.464360    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-brqnt"]
Jul 16 22:30:16.771251 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771164    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:16.771515 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771502    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771516    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771527    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771532    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771540    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771545    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.771559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771558    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.771742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771563    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.771742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771614    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.771742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771622    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771629    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:16.771742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.771635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:16.775998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.775982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:16.796562 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.796537    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:16.828497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.828473    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj\" (UID: \"c8c623fe-0712-45fc-95eb-74f916c52cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:16.929183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.929156    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj\" (UID: \"c8c623fe-0712-45fc-95eb-74f916c52cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:16.929700 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:16.929683    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj\" (UID: \"c8c623fe-0712-45fc-95eb-74f916c52cfd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:17.085720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.085644    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:17.160031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.159987    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" path="/var/lib/kubelet/pods/7ffaf08d-9fee-4cb7-a3b0-ccb039603364/volumes"
Jul 16 22:30:17.228097 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.228065    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:17.231034 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:30:17.231006    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c623fe_0712_45fc_95eb_74f916c52cfd.slice/crio-6d584bb4adcd69777b9eb8573f35dc3c5bb5c156107eac3ad724fc07dd6abf6b WatchSource:0}: Error finding container 6d584bb4adcd69777b9eb8573f35dc3c5bb5c156107eac3ad724fc07dd6abf6b: Status 404 returned error can't find the container with id 6d584bb4adcd69777b9eb8573f35dc3c5bb5c156107eac3ad724fc07dd6abf6b
Jul 16 22:30:17.381096 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.381064    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" event={"ID":"c8c623fe-0712-45fc-95eb-74f916c52cfd","Type":"ContainerStarted","Data":"6d584bb4adcd69777b9eb8573f35dc3c5bb5c156107eac3ad724fc07dd6abf6b"}
Jul 16 22:30:17.429768 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.429737    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:17.557553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:17.557532    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"
Jul 16 22:30:18.388189 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.388155    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"
Jul 16 22:30:18.388406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.388154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp" event={"ID":"9bce9913-1cdd-4c54-b7da-a526d0c6ea42","Type":"ContainerDied","Data":"cbf3a105f4251920095d4997ca737ba08aafb04ccc2c4ec50e6b819ef05eda8d"}
Jul 16 22:30:18.388406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.388292    2581 scope.go:117] "RemoveContainer" containerID="a754d64b9825217e5f79a295e7455763cd15af1183eaa99f4fbffe8e1ed58ee8"
Jul 16 22:30:18.389747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.389725    2581 generic.go:356] "Generic (PLEG): container finished" podID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerID="9e6b4625144ff045d00a619c4aa215b7f5e7164df186e38f218d538588a00431" exitCode=6
Jul 16 22:30:18.389861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.389799    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" event={"ID":"c8c623fe-0712-45fc-95eb-74f916c52cfd","Type":"ContainerDied","Data":"9e6b4625144ff045d00a619c4aa215b7f5e7164df186e38f218d538588a00431"}
Jul 16 22:30:18.390000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.389986    2581 scope.go:117] "RemoveContainer" containerID="9e6b4625144ff045d00a619c4aa215b7f5e7164df186e38f218d538588a00431"
Jul 16 22:30:18.455296 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.455272    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:18.468806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:18.468783    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-derive-a6237863-29737350-96bwp"]
Jul 16 22:30:19.160470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:19.160439    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" path="/var/lib/kubelet/pods/9bce9913-1cdd-4c54-b7da-a526d0c6ea42/volumes"
Jul 16 22:30:19.396491 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:19.396456    2581 generic.go:356] "Generic (PLEG): container finished" podID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerID="0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915" exitCode=6
Jul 16 22:30:19.396651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:19.396544    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" event={"ID":"c8c623fe-0712-45fc-95eb-74f916c52cfd","Type":"ContainerDied","Data":"0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915"}
Jul 16 22:30:19.396651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:19.396599    2581 scope.go:117] "RemoveContainer" containerID="9e6b4625144ff045d00a619c4aa215b7f5e7164df186e38f218d538588a00431"
Jul 16 22:30:19.396736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:19.396682    2581 scope.go:117] "RemoveContainer" containerID="0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915"
Jul 16 22:30:19.396891 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:19.396869    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj_odh-ai-gateway-infra(c8c623fe-0712-45fc-95eb-74f916c52cfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd"
Jul 16 22:30:20.402962 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:20.402934    2581 scope.go:117] "RemoveContainer" containerID="0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915"
Jul 16 22:30:20.403467 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:20.403182    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj_odh-ai-gateway-infra(c8c623fe-0712-45fc-95eb-74f916c52cfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd"
Jul 16 22:30:22.155384 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:22.155345    2581 scope.go:117] "RemoveContainer" containerID="3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162"
Jul 16 22:30:22.415183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:22.415105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerStarted","Data":"33ee022708289226500ea26f367fcd61c985c4db6fd293a0b5b73bd3d971aacb"}
Jul 16 22:30:22.431765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:22.431717    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" podStartSLOduration=17.431703077 podStartE2EDuration="17.431703077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:22.431156349 +0000 UTC m=+2609.760596250" watchObservedRunningTime="2026-07-16 22:30:22.431703077 +0000 UTC m=+2609.761142978"
Jul 16 22:30:23.420711 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:23.420679    2581 generic.go:356] "Generic (PLEG): container finished" podID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerID="33ee022708289226500ea26f367fcd61c985c4db6fd293a0b5b73bd3d971aacb" exitCode=6
Jul 16 22:30:23.421123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:23.420749    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerDied","Data":"33ee022708289226500ea26f367fcd61c985c4db6fd293a0b5b73bd3d971aacb"}
Jul 16 22:30:23.421123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:23.420813    2581 scope.go:117] "RemoveContainer" containerID="3ef24acfee48e5d7d65d15c8c3c36f6c3c2d23a942f8adf01018e311dd470162"
Jul 16 22:30:23.421123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:23.421111    2581 scope.go:117] "RemoveContainer" containerID="33ee022708289226500ea26f367fcd61c985c4db6fd293a0b5b73bd3d971aacb"
Jul 16 22:30:23.421346 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:23.421324    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj_odh-ai-gateway-infra(60a72c95-49b7-4c31-8668-82e8ecb889e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8"
Jul 16 22:30:23.448949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:23.448922    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:24.559573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:24.559550    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:24.691456 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:24.691372    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca\") pod \"60a72c95-49b7-4c31-8668-82e8ecb889e8\" (UID: \"60a72c95-49b7-4c31-8668-82e8ecb889e8\") "
Jul 16 22:30:24.691699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:24.691678    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "60a72c95-49b7-4c31-8668-82e8ecb889e8" (UID: "60a72c95-49b7-4c31-8668-82e8ecb889e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:24.792567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:24.792537    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/60a72c95-49b7-4c31-8668-82e8ecb889e8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:30:25.433047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:25.432959    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj" event={"ID":"60a72c95-49b7-4c31-8668-82e8ecb889e8","Type":"ContainerDied","Data":"801f47cb16cf36d2685faf9af71d5cb5ebda14071edccd0c34c8a59e7a250b14"}
Jul 16 22:30:25.433047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:25.433027    2581 scope.go:117] "RemoveContainer" containerID="33ee022708289226500ea26f367fcd61c985c4db6fd293a0b5b73bd3d971aacb"
Jul 16 22:30:25.433315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:25.432969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"
Jul 16 22:30:25.457050 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:25.457021    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:25.466184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:25.466162    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-rhtcj"]
Jul 16 22:30:27.159995 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:27.159962    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" path="/var/lib/kubelet/pods/60a72c95-49b7-4c31-8668-82e8ecb889e8/volumes"
Jul 16 22:30:32.154847 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:32.154815    2581 scope.go:117] "RemoveContainer" containerID="0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915"
Jul 16 22:30:33.472107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:33.472071    2581 generic.go:356] "Generic (PLEG): container finished" podID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerID="7c1c66ab1be0356e1d84da20fd40b1e18bde45709f03224b08e6fa405ff5ca92" exitCode=6
Jul 16 22:30:33.472611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:33.472120    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" event={"ID":"c8c623fe-0712-45fc-95eb-74f916c52cfd","Type":"ContainerDied","Data":"7c1c66ab1be0356e1d84da20fd40b1e18bde45709f03224b08e6fa405ff5ca92"}
Jul 16 22:30:33.472611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:33.472169    2581 scope.go:117] "RemoveContainer" containerID="0cb9da2e5b3bb4e437483eccb625dce48836578b4e8820306a3c23296814c915"
Jul 16 22:30:33.472611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:33.472494    2581 scope.go:117] "RemoveContainer" containerID="7c1c66ab1be0356e1d84da20fd40b1e18bde45709f03224b08e6fa405ff5ca92"
Jul 16 22:30:33.472771 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:33.472743    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj_odh-ai-gateway-infra(c8c623fe-0712-45fc-95eb-74f916c52cfd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd"
Jul 16 22:30:34.521565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:34.521531    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:34.665722 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:34.665697    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:34.777865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:34.777789    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca\") pod \"c8c623fe-0712-45fc-95eb-74f916c52cfd\" (UID: \"c8c623fe-0712-45fc-95eb-74f916c52cfd\") "
Jul 16 22:30:34.778129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:34.778108    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8c623fe-0712-45fc-95eb-74f916c52cfd" (UID: "c8c623fe-0712-45fc-95eb-74f916c52cfd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:34.878964 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:34.878930    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c623fe-0712-45fc-95eb-74f916c52cfd-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:30:35.483297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:35.483196    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"
Jul 16 22:30:35.483297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:35.483202    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj" event={"ID":"c8c623fe-0712-45fc-95eb-74f916c52cfd","Type":"ContainerDied","Data":"6d584bb4adcd69777b9eb8573f35dc3c5bb5c156107eac3ad724fc07dd6abf6b"}
Jul 16 22:30:35.483524 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:35.483302    2581 scope.go:117] "RemoveContainer" containerID="7c1c66ab1be0356e1d84da20fd40b1e18bde45709f03224b08e6fa405ff5ca92"
Jul 16 22:30:35.517291 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:35.517265    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:35.526571 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:35.526552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-nvrqj"]
Jul 16 22:30:37.160444 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:37.160412    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" path="/var/lib/kubelet/pods/c8c623fe-0712-45fc-95eb-74f916c52cfd/volumes"
Jul 16 22:30:42.883571 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883538    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883856    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883866    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883876    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883883    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883889    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883895    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883903    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883908    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883914    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883919    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883924    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883929    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883940    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883945    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883951    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883955    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883960    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.883966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.883965    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884015    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884021    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884026    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffaf08d-9fee-4cb7-a3b0-ccb039603364" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884031    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884037    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884042    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884046    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884052    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9bce9913-1cdd-4c54-b7da-a526d0c6ea42" containerName="cleanup"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884056    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="add2fd15-03a4-4c63-ad18-a9a3eeb082f2" containerName="revoke-keys"
Jul 16 22:30:42.884640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.884064    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:42.888304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.888288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:30:42.899239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.899211    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:30:42.899555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.899541    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:30:42.925854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:42.925831    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:30:43.046048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.046020    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-22xzz\" (UID: \"3497ae11-9406-4542-94b6-b40f10042707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:30:43.146925 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.146851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-22xzz\" (UID: \"3497ae11-9406-4542-94b6-b40f10042707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:30:43.147470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.147452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-22xzz\" (UID: \"3497ae11-9406-4542-94b6-b40f10042707\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:30:43.198139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.198117    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:30:43.342822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.342777    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:30:43.346328 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:30:43.346298    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3497ae11_9406_4542_94b6_b40f10042707.slice/crio-7070c326b61ab4ec582c73696fc4049358f815e24d3cc2fa50c71f7fe2a56e41 WatchSource:0}: Error finding container 7070c326b61ab4ec582c73696fc4049358f815e24d3cc2fa50c71f7fe2a56e41: Status 404 returned error can't find the container with id 7070c326b61ab4ec582c73696fc4049358f815e24d3cc2fa50c71f7fe2a56e41
Jul 16 22:30:43.525905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:43.525819    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerStarted","Data":"7070c326b61ab4ec582c73696fc4049358f815e24d3cc2fa50c71f7fe2a56e41"}
Jul 16 22:30:44.532320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:44.532288    2581 generic.go:356] "Generic (PLEG): container finished" podID="3497ae11-9406-4542-94b6-b40f10042707" containerID="251eb541b109d43b99f302a4c49ac257710e73cc8befa3c85c9fed044cbf6ac7" exitCode=7
Jul 16 22:30:44.532709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:44.532386    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerDied","Data":"251eb541b109d43b99f302a4c49ac257710e73cc8befa3c85c9fed044cbf6ac7"}
Jul 16 22:30:44.532709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:44.532528    2581 scope.go:117] "RemoveContainer" containerID="251eb541b109d43b99f302a4c49ac257710e73cc8befa3c85c9fed044cbf6ac7"
Jul 16 22:30:45.538757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:45.538668    2581 generic.go:356] "Generic (PLEG): container finished" podID="3497ae11-9406-4542-94b6-b40f10042707" containerID="ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404" exitCode=7
Jul 16 22:30:45.538757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:45.538738    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerDied","Data":"ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404"}
Jul 16 22:30:45.539273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:45.538783    2581 scope.go:117] "RemoveContainer" containerID="251eb541b109d43b99f302a4c49ac257710e73cc8befa3c85c9fed044cbf6ac7"
Jul 16 22:30:45.539273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:45.539043    2581 scope.go:117] "RemoveContainer" containerID="ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404"
Jul 16 22:30:45.539399 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:45.539308    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-22xzz_odh-ai-gateway-infra(3497ae11-9406-4542-94b6-b40f10042707)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" podUID="3497ae11-9406-4542-94b6-b40f10042707"
Jul 16 22:30:46.545187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:46.545161    2581 scope.go:117] "RemoveContainer" containerID="ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404"
Jul 16 22:30:46.545579 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:46.545342    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-22xzz_odh-ai-gateway-infra(3497ae11-9406-4542-94b6-b40f10042707)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" podUID="3497ae11-9406-4542-94b6-b40f10042707"
Jul 16 22:30:54.405308 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405274    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:30:54.405710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405606    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:54.405710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405618    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="60a72c95-49b7-4c31-8668-82e8ecb889e8" containerName="revoke-keys"
Jul 16 22:30:54.405710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405628    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:54.405710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405633    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:54.405710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.405690    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c623fe-0712-45fc-95eb-74f916c52cfd" containerName="revoke-keys"
Jul 16 22:30:54.410067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.410049    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:30:54.417319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.416906    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:30:54.426656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.426629    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-6m726\" (UID: \"38b2c349-f668-4f39-93f3-0ac46397152f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:30:54.527909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.527871    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-6m726\" (UID: \"38b2c349-f668-4f39-93f3-0ac46397152f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:30:54.528548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.528526    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-6m726\" (UID: \"38b2c349-f668-4f39-93f3-0ac46397152f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:30:54.721717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.721638    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:30:54.852256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:54.852146    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:30:54.854467 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:30:54.854436    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38b2c349_f668_4f39_93f3_0ac46397152f.slice/crio-533dc0e90ee3cd864825b889dca4a7ad2deda427c2302b6d5ec6ebc366b4905b WatchSource:0}: Error finding container 533dc0e90ee3cd864825b889dca4a7ad2deda427c2302b6d5ec6ebc366b4905b: Status 404 returned error can't find the container with id 533dc0e90ee3cd864825b889dca4a7ad2deda427c2302b6d5ec6ebc366b4905b
Jul 16 22:30:55.583838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:55.583802    2581 generic.go:356] "Generic (PLEG): container finished" podID="38b2c349-f668-4f39-93f3-0ac46397152f" containerID="05be7c35b11b2f0eeffbae8f437f6239b0cbd4b769fce002d252412d0117d86f" exitCode=6
Jul 16 22:30:55.584266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:55.583877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerDied","Data":"05be7c35b11b2f0eeffbae8f437f6239b0cbd4b769fce002d252412d0117d86f"}
Jul 16 22:30:55.584266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:55.583933    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerStarted","Data":"533dc0e90ee3cd864825b889dca4a7ad2deda427c2302b6d5ec6ebc366b4905b"}
Jul 16 22:30:55.584266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:55.584114    2581 scope.go:117] "RemoveContainer" containerID="05be7c35b11b2f0eeffbae8f437f6239b0cbd4b769fce002d252412d0117d86f"
Jul 16 22:30:56.590608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:56.590574    2581 generic.go:356] "Generic (PLEG): container finished" podID="38b2c349-f668-4f39-93f3-0ac46397152f" containerID="2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842" exitCode=6
Jul 16 22:30:56.590992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:56.590657    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerDied","Data":"2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842"}
Jul 16 22:30:56.590992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:56.590710    2581 scope.go:117] "RemoveContainer" containerID="05be7c35b11b2f0eeffbae8f437f6239b0cbd4b769fce002d252412d0117d86f"
Jul 16 22:30:56.590992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:56.590840    2581 scope.go:117] "RemoveContainer" containerID="2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842"
Jul 16 22:30:56.591119 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:56.591083    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-6m726_odh-ai-gateway-infra(38b2c349-f668-4f39-93f3-0ac46397152f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" podUID="38b2c349-f668-4f39-93f3-0ac46397152f"
Jul 16 22:30:57.596783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:57.596757    2581 scope.go:117] "RemoveContainer" containerID="2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842"
Jul 16 22:30:57.597174 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:57.596948    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-6m726_odh-ai-gateway-infra(38b2c349-f668-4f39-93f3-0ac46397152f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" podUID="38b2c349-f668-4f39-93f3-0ac46397152f"
Jul 16 22:30:59.155423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:59.155395    2581 scope.go:117] "RemoveContainer" containerID="ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404"
Jul 16 22:30:59.606101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:59.606070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerStarted","Data":"b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611"}
Jul 16 22:30:59.623277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:30:59.623216    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" podStartSLOduration=17.623201598 podStartE2EDuration="17.623201598s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:59.621577656 +0000 UTC m=+2646.951017550" watchObservedRunningTime="2026-07-16 22:30:59.623201598 +0000 UTC m=+2646.952641499"
Jul 16 22:30:59.786502 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:30:59.786472    2581 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3497ae11_9406_4542_94b6_b40f10042707.slice/crio-conmon-b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:31:00.613063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:00.613031    2581 generic.go:356] "Generic (PLEG): container finished" podID="3497ae11-9406-4542-94b6-b40f10042707" containerID="b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611" exitCode=7
Jul 16 22:31:00.613550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:00.613105    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerDied","Data":"b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611"}
Jul 16 22:31:00.613550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:00.613164    2581 scope.go:117] "RemoveContainer" containerID="ab07b5eac8e101089bfb16d4659d70bad9d518f8c3ef9d16d9931419c6919404"
Jul 16 22:31:00.613550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:00.613427    2581 scope.go:117] "RemoveContainer" containerID="b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611"
Jul 16 22:31:00.613674 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:00.613613    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-22xzz_odh-ai-gateway-infra(3497ae11-9406-4542-94b6-b40f10042707)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" podUID="3497ae11-9406-4542-94b6-b40f10042707"
Jul 16 22:31:00.640877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:00.640851    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:31:01.748011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:01.747986    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:31:01.776106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:01.776083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca\") pod \"3497ae11-9406-4542-94b6-b40f10042707\" (UID: \"3497ae11-9406-4542-94b6-b40f10042707\") "
Jul 16 22:31:01.776397 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:01.776377    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3497ae11-9406-4542-94b6-b40f10042707" (UID: "3497ae11-9406-4542-94b6-b40f10042707"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:01.877154 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:01.877092    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3497ae11-9406-4542-94b6-b40f10042707-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:02.624716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.624691    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"
Jul 16 22:31:02.624894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.624693    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz" event={"ID":"3497ae11-9406-4542-94b6-b40f10042707","Type":"ContainerDied","Data":"7070c326b61ab4ec582c73696fc4049358f815e24d3cc2fa50c71f7fe2a56e41"}
Jul 16 22:31:02.624894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.624825    2581 scope.go:117] "RemoveContainer" containerID="b3f29faf3e3473ee00127aa4a4bdf5ab1cfda418d01a48d3a03927e987aa3611"
Jul 16 22:31:02.647136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.647108    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:31:02.651620 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.651595    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-22xzz"]
Jul 16 22:31:02.794381 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.794351    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:31:02.794704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:02.794539    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" containerID="cri-o://c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16" gracePeriod=30
Jul 16 22:31:03.160782 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:03.160746    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3497ae11-9406-4542-94b6-b40f10042707" path="/var/lib/kubelet/pods/3497ae11-9406-4542-94b6-b40f10042707/volumes"
Jul 16 22:31:05.273123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273094    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273478    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273490    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273511    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273516    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273524    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273530    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273576    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.273778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.273582    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:05.277559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.277538    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:05.293882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.293855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:05.300377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.300355    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz\" (UID: \"ae95e6cf-427b-4fa4-a582-a6b4655be8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:05.400708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.400679    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz\" (UID: \"ae95e6cf-427b-4fa4-a582-a6b4655be8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:05.401266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.401249    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz\" (UID: \"ae95e6cf-427b-4fa4-a582-a6b4655be8c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:05.434740 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.434714    2581 patch_prober.go:28] interesting pod/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:31:05.434852 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.434773    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:31:05.587485 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.587458    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:05.610224 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.610199    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:31:05.715419 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:05.715255    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:05.717409 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:31:05.717382    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae95e6cf_427b_4fa4_a582_a6b4655be8c3.slice/crio-30bf3edaa9187d0b8997b1cfaaff4449b2f32c28089c6404210ea5c19cc7479e WatchSource:0}: Error finding container 30bf3edaa9187d0b8997b1cfaaff4449b2f32c28089c6404210ea5c19cc7479e: Status 404 returned error can't find the container with id 30bf3edaa9187d0b8997b1cfaaff4449b2f32c28089c6404210ea5c19cc7479e
Jul 16 22:31:06.646518 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.646431    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerID="30768ec8aef07f146b35417c279272e49aa4cd1b75a67e0ab026a9a047dde1e4" exitCode=6
Jul 16 22:31:06.646904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.646518    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerDied","Data":"30768ec8aef07f146b35417c279272e49aa4cd1b75a67e0ab026a9a047dde1e4"}
Jul 16 22:31:06.646904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.646572    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerStarted","Data":"30bf3edaa9187d0b8997b1cfaaff4449b2f32c28089c6404210ea5c19cc7479e"}
Jul 16 22:31:06.646904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.646697    2581 scope.go:117] "RemoveContainer" containerID="30768ec8aef07f146b35417c279272e49aa4cd1b75a67e0ab026a9a047dde1e4"
Jul 16 22:31:06.813939 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.813903    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:06.814401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.814382    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3497ae11-9406-4542-94b6-b40f10042707" containerName="revoke-keys"
Jul 16 22:31:06.818310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.818291    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:06.828779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.828747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:06.911118 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:06.911054    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr\" (UID: \"f6e79fe0-09bb-4220-abc7-535c03016635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:07.012210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.012185    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr\" (UID: \"f6e79fe0-09bb-4220-abc7-535c03016635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:07.012721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.012702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr\" (UID: \"f6e79fe0-09bb-4220-abc7-535c03016635\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:07.127888 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.127863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:07.258067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.258038    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:07.258381 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:31:07.258356    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6e79fe0_09bb_4220_abc7_535c03016635.slice/crio-b4ced982ed0fbdbce80f2b68d27883c0e98fcb9fff07f48e02afb10e82c5480c WatchSource:0}: Error finding container b4ced982ed0fbdbce80f2b68d27883c0e98fcb9fff07f48e02afb10e82c5480c: Status 404 returned error can't find the container with id b4ced982ed0fbdbce80f2b68d27883c0e98fcb9fff07f48e02afb10e82c5480c
Jul 16 22:31:07.652181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.652153    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerID="23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de" exitCode=6
Jul 16 22:31:07.652560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.652255    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerDied","Data":"23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de"}
Jul 16 22:31:07.652560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.652315    2581 scope.go:117] "RemoveContainer" containerID="30768ec8aef07f146b35417c279272e49aa4cd1b75a67e0ab026a9a047dde1e4"
Jul 16 22:31:07.652560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.652515    2581 scope.go:117] "RemoveContainer" containerID="23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de"
Jul 16 22:31:07.652759 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:07.652734    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz_odh-ai-gateway-infra(ae95e6cf-427b-4fa4-a582-a6b4655be8c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3"
Jul 16 22:31:07.653730 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.653712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerStarted","Data":"20883cd87c0a9e125932517c8786fac090037662362bf498b70ed9110ae0d602"}
Jul 16 22:31:07.653813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.653755    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerStarted","Data":"b4ced982ed0fbdbce80f2b68d27883c0e98fcb9fff07f48e02afb10e82c5480c"}
Jul 16 22:31:07.690171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:07.690134    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" podStartSLOduration=1.690121303 podStartE2EDuration="1.690121303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:07.688586529 +0000 UTC m=+2655.018026454" watchObservedRunningTime="2026-07-16 22:31:07.690121303 +0000 UTC m=+2655.019561204"
Jul 16 22:31:08.050697 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.050672    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:31:08.119864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.119838    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.119889    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.119931    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120024    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120083    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120119    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8d4pw\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120122    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:08.120337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120154    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120195    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120271    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy\") pod \"24e42674-0b4d-47be-8eeb-c9a9919720da\" (UID: \"24e42674-0b4d-47be-8eeb-c9a9919720da\") "
Jul 16 22:31:08.120518 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120396    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:08.120572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120551    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.120572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120567    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24e42674-0b4d-47be-8eeb-c9a9919720da-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.120685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120632    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:08.120685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120642    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data" (OuterVolumeSpecName: "istio-data") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:08.120795 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.120702    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:08.122841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.122816    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token" (OuterVolumeSpecName: "istio-token") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:08.122841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.122814    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:31:08.122991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.122865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw" (OuterVolumeSpecName: "kube-api-access-8d4pw") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "kube-api-access-8d4pw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:31:08.123088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.123069    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "24e42674-0b4d-47be-8eeb-c9a9919720da" (UID: "24e42674-0b4d-47be-8eeb-c9a9919720da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:31:08.221322 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221296    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221322 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221322    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221334    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221344    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24e42674-0b4d-47be-8eeb-c9a9919720da-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221352    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8d4pw\" (UniqueName: \"kubernetes.io/projected/24e42674-0b4d-47be-8eeb-c9a9919720da-kube-api-access-8d4pw\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221361    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.221494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.221368    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24e42674-0b4d-47be-8eeb-c9a9919720da-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:08.660594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.660561    2581 scope.go:117] "RemoveContainer" containerID="23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de"
Jul 16 22:31:08.661049 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:08.660762    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz_odh-ai-gateway-infra(ae95e6cf-427b-4fa4-a582-a6b4655be8c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3"
Jul 16 22:31:08.662595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.662352    2581 generic.go:356] "Generic (PLEG): container finished" podID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerID="c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16" exitCode=0
Jul 16 22:31:08.662595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.662392    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerDied","Data":"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"}
Jul 16 22:31:08.662595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.662446    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r" event={"ID":"24e42674-0b4d-47be-8eeb-c9a9919720da","Type":"ContainerDied","Data":"a5c874529b4182f75a4d36375e5bbdb44d98494e188a6ac1ad7c03af780f4a64"}
Jul 16 22:31:08.662595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.662405    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"
Jul 16 22:31:08.662595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.662472    2581 scope.go:117] "RemoveContainer" containerID="c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"
Jul 16 22:31:08.666064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.665183    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e79fe0-09bb-4220-abc7-535c03016635" containerID="20883cd87c0a9e125932517c8786fac090037662362bf498b70ed9110ae0d602" exitCode=6
Jul 16 22:31:08.666064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.665398    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerDied","Data":"20883cd87c0a9e125932517c8786fac090037662362bf498b70ed9110ae0d602"}
Jul 16 22:31:08.666064 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.665585    2581 scope.go:117] "RemoveContainer" containerID="20883cd87c0a9e125932517c8786fac090037662362bf498b70ed9110ae0d602"
Jul 16 22:31:08.676008 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.675988    2581 scope.go:117] "RemoveContainer" containerID="cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"
Jul 16 22:31:08.688751 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.688735    2581 scope.go:117] "RemoveContainer" containerID="c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"
Jul 16 22:31:08.688982 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:08.688963    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16\": container with ID starting with c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16 not found: ID does not exist" containerID="c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"
Jul 16 22:31:08.689042 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.688992    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16"} err="failed to get container status \"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16\": rpc error: code = NotFound desc = could not find container \"c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16\": container with ID starting with c60330397855041b1a20c9b0630cd9880a46741140432449b38ff4c7b2d7ed16 not found: ID does not exist"
Jul 16 22:31:08.689042 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.689009    2581 scope.go:117] "RemoveContainer" containerID="cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"
Jul 16 22:31:08.689284 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:08.689266    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c\": container with ID starting with cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c not found: ID does not exist" containerID="cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"
Jul 16 22:31:08.689345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.689290    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c"} err="failed to get container status \"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c\": rpc error: code = NotFound desc = could not find container \"cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c\": container with ID starting with cec9615854cce9c0daea1e2ae0728730a265a9f5d75fda56ec7e50041f93ad2c not found: ID does not exist"
Jul 16 22:31:08.694849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.694828    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:31:08.699677 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:08.699655    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a6237863-openshift-default-5dddfc467f-zht6r"]
Jul 16 22:31:09.162055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:09.162013    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" path="/var/lib/kubelet/pods/24e42674-0b4d-47be-8eeb-c9a9919720da/volumes"
Jul 16 22:31:09.671972 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:09.671934    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e79fe0-09bb-4220-abc7-535c03016635" containerID="9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d" exitCode=6
Jul 16 22:31:09.672371 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:09.672022    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerDied","Data":"9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d"}
Jul 16 22:31:09.672371 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:09.672080    2581 scope.go:117] "RemoveContainer" containerID="20883cd87c0a9e125932517c8786fac090037662362bf498b70ed9110ae0d602"
Jul 16 22:31:09.672371 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:09.672288    2581 scope.go:117] "RemoveContainer" containerID="9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d"
Jul 16 22:31:09.672546 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:09.672472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr_odh-ai-gateway-infra(f6e79fe0-09bb-4220-abc7-535c03016635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" podUID="f6e79fe0-09bb-4220-abc7-535c03016635"
Jul 16 22:31:10.154801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:10.154771    2581 scope.go:117] "RemoveContainer" containerID="2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842"
Jul 16 22:31:10.678043 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:10.678007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerStarted","Data":"2dc06db05acf4edfaa56fed87a03ad75b220f84ba1aae966296b3148084dcfa1"}
Jul 16 22:31:10.679568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:10.679553    2581 scope.go:117] "RemoveContainer" containerID="9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d"
Jul 16 22:31:10.679715 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:10.679697    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr_odh-ai-gateway-infra(f6e79fe0-09bb-4220-abc7-535c03016635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" podUID="f6e79fe0-09bb-4220-abc7-535c03016635"
Jul 16 22:31:10.694569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:10.694525    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" podStartSLOduration=16.694512803 podStartE2EDuration="16.694512803s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:10.692944665 +0000 UTC m=+2658.022384567" watchObservedRunningTime="2026-07-16 22:31:10.694512803 +0000 UTC m=+2658.023952707"
Jul 16 22:31:11.685471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:11.685434    2581 generic.go:356] "Generic (PLEG): container finished" podID="38b2c349-f668-4f39-93f3-0ac46397152f" containerID="2dc06db05acf4edfaa56fed87a03ad75b220f84ba1aae966296b3148084dcfa1" exitCode=6
Jul 16 22:31:11.685908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:11.685508    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerDied","Data":"2dc06db05acf4edfaa56fed87a03ad75b220f84ba1aae966296b3148084dcfa1"}
Jul 16 22:31:11.685908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:11.685569    2581 scope.go:117] "RemoveContainer" containerID="2f875ff6c8bbea0983f7455c4fdef0ba323451af2a412411da927b181bb36842"
Jul 16 22:31:11.685908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:11.685808    2581 scope.go:117] "RemoveContainer" containerID="2dc06db05acf4edfaa56fed87a03ad75b220f84ba1aae966296b3148084dcfa1"
Jul 16 22:31:11.686031 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:11.686002    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-6m726_odh-ai-gateway-infra(38b2c349-f668-4f39-93f3-0ac46397152f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" podUID="38b2c349-f668-4f39-93f3-0ac46397152f"
Jul 16 22:31:11.709019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:11.708993    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:31:12.814723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:12.814700    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:31:12.858845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:12.858812    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca\") pod \"38b2c349-f668-4f39-93f3-0ac46397152f\" (UID: \"38b2c349-f668-4f39-93f3-0ac46397152f\") "
Jul 16 22:31:12.859129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:12.859109    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38b2c349-f668-4f39-93f3-0ac46397152f" (UID: "38b2c349-f668-4f39-93f3-0ac46397152f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:12.959581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:12.959508    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38b2c349-f668-4f39-93f3-0ac46397152f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:13.696738 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:13.696708    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"
Jul 16 22:31:13.696738 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:13.696729    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726" event={"ID":"38b2c349-f668-4f39-93f3-0ac46397152f","Type":"ContainerDied","Data":"533dc0e90ee3cd864825b889dca4a7ad2deda427c2302b6d5ec6ebc366b4905b"}
Jul 16 22:31:13.697039 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:13.696802    2581 scope.go:117] "RemoveContainer" containerID="2dc06db05acf4edfaa56fed87a03ad75b220f84ba1aae966296b3148084dcfa1"
Jul 16 22:31:13.719195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:13.719170    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:31:13.729989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:13.729964    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-6m726"]
Jul 16 22:31:15.159644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:15.159610    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" path="/var/lib/kubelet/pods/38b2c349-f668-4f39-93f3-0ac46397152f/volumes"
Jul 16 22:31:20.154669 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:20.154633    2581 scope.go:117] "RemoveContainer" containerID="23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de"
Jul 16 22:31:20.730871 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:20.730796    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerStarted","Data":"ff6fae85b7ac412c31d8b2b9d0c70eb1991e7fc6db45453a0ed59b0888de98b6"}
Jul 16 22:31:20.788214 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:20.788169    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" podStartSLOduration=15.788153601 podStartE2EDuration="15.788153601s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:20.787299395 +0000 UTC m=+2668.116739297" watchObservedRunningTime="2026-07-16 22:31:20.788153601 +0000 UTC m=+2668.117593502"
Jul 16 22:31:21.736856 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:21.736823    2581 generic.go:356] "Generic (PLEG): container finished" podID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerID="ff6fae85b7ac412c31d8b2b9d0c70eb1991e7fc6db45453a0ed59b0888de98b6" exitCode=6
Jul 16 22:31:21.737263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:21.736877    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerDied","Data":"ff6fae85b7ac412c31d8b2b9d0c70eb1991e7fc6db45453a0ed59b0888de98b6"}
Jul 16 22:31:21.737263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:21.736927    2581 scope.go:117] "RemoveContainer" containerID="23b02acb8790d2297dc94069df4836e17b64379802e48483c6d6adda206562de"
Jul 16 22:31:21.737263 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:21.737160    2581 scope.go:117] "RemoveContainer" containerID="ff6fae85b7ac412c31d8b2b9d0c70eb1991e7fc6db45453a0ed59b0888de98b6"
Jul 16 22:31:21.737429 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:21.737404    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz_odh-ai-gateway-infra(ae95e6cf-427b-4fa4-a582-a6b4655be8c3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3"
Jul 16 22:31:21.804636 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:21.804600    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:22.872393 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:22.872368    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:22.934239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:22.934210    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca\") pod \"ae95e6cf-427b-4fa4-a582-a6b4655be8c3\" (UID: \"ae95e6cf-427b-4fa4-a582-a6b4655be8c3\") "
Jul 16 22:31:22.934568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:22.934550    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae95e6cf-427b-4fa4-a582-a6b4655be8c3" (UID: "ae95e6cf-427b-4fa4-a582-a6b4655be8c3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:23.034906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.034835    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae95e6cf-427b-4fa4-a582-a6b4655be8c3-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:23.748600 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.748572    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"
Jul 16 22:31:23.748748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.748594    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz" event={"ID":"ae95e6cf-427b-4fa4-a582-a6b4655be8c3","Type":"ContainerDied","Data":"30bf3edaa9187d0b8997b1cfaaff4449b2f32c28089c6404210ea5c19cc7479e"}
Jul 16 22:31:23.748748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.748656    2581 scope.go:117] "RemoveContainer" containerID="ff6fae85b7ac412c31d8b2b9d0c70eb1991e7fc6db45453a0ed59b0888de98b6"
Jul 16 22:31:23.767998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.767978    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:23.772647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:23.772628    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-htdnz"]
Jul 16 22:31:24.154850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:24.154820    2581 scope.go:117] "RemoveContainer" containerID="9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d"
Jul 16 22:31:24.754765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:24.754728    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerStarted","Data":"f16cea00aef84028306812c66d760f9bde4910ed842e9864d20581ed10757190"}
Jul 16 22:31:25.160635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.160602    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" path="/var/lib/kubelet/pods/ae95e6cf-427b-4fa4-a582-a6b4655be8c3/volumes"
Jul 16 22:31:25.761697 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.761666    2581 generic.go:356] "Generic (PLEG): container finished" podID="f6e79fe0-09bb-4220-abc7-535c03016635" containerID="f16cea00aef84028306812c66d760f9bde4910ed842e9864d20581ed10757190" exitCode=6
Jul 16 22:31:25.761874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.761712    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerDied","Data":"f16cea00aef84028306812c66d760f9bde4910ed842e9864d20581ed10757190"}
Jul 16 22:31:25.761874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.761759    2581 scope.go:117] "RemoveContainer" containerID="9a16dbca36566cf0389c527ec37422f4397ebce090b2889cd64d675be5ec914d"
Jul 16 22:31:25.762055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.762033    2581 scope.go:117] "RemoveContainer" containerID="f16cea00aef84028306812c66d760f9bde4910ed842e9864d20581ed10757190"
Jul 16 22:31:25.762315 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:25.762268    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr_odh-ai-gateway-infra(f6e79fe0-09bb-4220-abc7-535c03016635)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" podUID="f6e79fe0-09bb-4220-abc7-535c03016635"
Jul 16 22:31:25.794777 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:25.794748    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:26.896599 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:26.896573    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:26.965370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:26.965341    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca\") pod \"f6e79fe0-09bb-4220-abc7-535c03016635\" (UID: \"f6e79fe0-09bb-4220-abc7-535c03016635\") "
Jul 16 22:31:26.965694 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:26.965673    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f6e79fe0-09bb-4220-abc7-535c03016635" (UID: "f6e79fe0-09bb-4220-abc7-535c03016635"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:27.066396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.066322    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f6e79fe0-09bb-4220-abc7-535c03016635-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:27.773065 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.773025    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr" event={"ID":"f6e79fe0-09bb-4220-abc7-535c03016635","Type":"ContainerDied","Data":"b4ced982ed0fbdbce80f2b68d27883c0e98fcb9fff07f48e02afb10e82c5480c"}
Jul 16 22:31:27.773065 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.773062    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"
Jul 16 22:31:27.773364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.773091    2581 scope.go:117] "RemoveContainer" containerID="f16cea00aef84028306812c66d760f9bde4910ed842e9864d20581ed10757190"
Jul 16 22:31:27.801243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.801201    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:27.826736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:27.826711    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mctnr"]
Jul 16 22:31:29.160907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:29.160877    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" path="/var/lib/kubelet/pods/f6e79fe0-09bb-4220-abc7-535c03016635/volumes"
Jul 16 22:31:32.959076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959038    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959415    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959428    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959442    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959448    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959455    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959460    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959466    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959471    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959476    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959480    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959488    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959493    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959499    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959504    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959512    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959517    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959525    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959530    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959579    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959586    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959591    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959596    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959601    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959606    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959612    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959617    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:32.959919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.959623    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:32.962239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.962215    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:32.965044 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.965022    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:31:32.967590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.967573    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:31:32.978417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:32.978394    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:33.008166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.008136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7pkq6\" (UID: \"83eb70f7-be19-43f0-b749-f574218ef9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:33.108840 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.108806    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7pkq6\" (UID: \"83eb70f7-be19-43f0-b749-f574218ef9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:33.109444 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.109425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-7pkq6\" (UID: \"83eb70f7-be19-43f0-b749-f574218ef9ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:33.272168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.272075    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:33.425380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.425357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:33.427974 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:31:33.427946    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83eb70f7_be19_43f0_b749_f574218ef9ed.slice/crio-ba382856e378f397829c793713869668ed554e2ac66ca0f103f24c256874a2c1 WatchSource:0}: Error finding container ba382856e378f397829c793713869668ed554e2ac66ca0f103f24c256874a2c1: Status 404 returned error can't find the container with id ba382856e378f397829c793713869668ed554e2ac66ca0f103f24c256874a2c1
Jul 16 22:31:33.801565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.801527    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerStarted","Data":"af3ffa591ed2726baefa5548aa72f2aa69b8239e4109e1080334fab7685ff5bb"}
Jul 16 22:31:33.801727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.801588    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerStarted","Data":"ba382856e378f397829c793713869668ed554e2ac66ca0f103f24c256874a2c1"}
Jul 16 22:31:33.830890 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:33.830842    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" podStartSLOduration=1.830828806 podStartE2EDuration="1.830828806s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:31:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:33.829916639 +0000 UTC m=+2681.159356540" watchObservedRunningTime="2026-07-16 22:31:33.830828806 +0000 UTC m=+2681.160268708"
Jul 16 22:31:34.807885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:34.807844    2581 generic.go:356] "Generic (PLEG): container finished" podID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerID="af3ffa591ed2726baefa5548aa72f2aa69b8239e4109e1080334fab7685ff5bb" exitCode=6
Jul 16 22:31:34.808325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:34.807938    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerDied","Data":"af3ffa591ed2726baefa5548aa72f2aa69b8239e4109e1080334fab7685ff5bb"}
Jul 16 22:31:34.808325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:34.808090    2581 scope.go:117] "RemoveContainer" containerID="af3ffa591ed2726baefa5548aa72f2aa69b8239e4109e1080334fab7685ff5bb"
Jul 16 22:31:35.813607 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:35.813523    2581 generic.go:356] "Generic (PLEG): container finished" podID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerID="08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff" exitCode=6
Jul 16 22:31:35.813607 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:35.813571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerDied","Data":"08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff"}
Jul 16 22:31:35.814023 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:35.813618    2581 scope.go:117] "RemoveContainer" containerID="af3ffa591ed2726baefa5548aa72f2aa69b8239e4109e1080334fab7685ff5bb"
Jul 16 22:31:35.814023 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:35.813775    2581 scope.go:117] "RemoveContainer" containerID="08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff"
Jul 16 22:31:35.814023 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:35.813981    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7pkq6_odh-ai-gateway-infra(83eb70f7-be19-43f0-b749-f574218ef9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed"
Jul 16 22:31:36.819789 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:36.819762    2581 scope.go:117] "RemoveContainer" containerID="08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff"
Jul 16 22:31:36.820190 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:36.819944    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7pkq6_odh-ai-gateway-infra(83eb70f7-be19-43f0-b749-f574218ef9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed"
Jul 16 22:31:44.487310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487219    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487595    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487611    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="38b2c349-f668-4f39-93f3-0ac46397152f" containerName="revoke-keys"
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487636    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487642    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6e79fe0-09bb-4220-abc7-535c03016635" containerName="revoke-keys"
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487695    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae95e6cf-427b-4fa4-a582-a6b4655be8c3" containerName="revoke-keys"
Jul 16 22:31:44.487725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.487707    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="24e42674-0b4d-47be-8eeb-c9a9919720da" containerName="istio-proxy"
Jul 16 22:31:44.491798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.491778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.495370 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:44.495336    2581 status_manager.go:1018] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb\" is forbidden: User \"system:node:ip-10-0-138-223.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-138-223.ec2.internal' and this object" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.510137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.510110    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:31:44.598849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.598808    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb\" (UID: \"87a4c1ae-3507-46e2-b445-107b0bd2b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.700333 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.700293    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb\" (UID: \"87a4c1ae-3507-46e2-b445-107b0bd2b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.700931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.700910    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb\" (UID: \"87a4c1ae-3507-46e2-b445-107b0bd2b0fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.801298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.801164    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:31:44.945214 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:44.945184    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:31:44.947916 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:31:44.947887    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87a4c1ae_3507_46e2_b445_107b0bd2b0fe.slice/crio-d9bd92bb60e2b204b82aca0493cafb79d82da5c05b00a0800da6974314268c61 WatchSource:0}: Error finding container d9bd92bb60e2b204b82aca0493cafb79d82da5c05b00a0800da6974314268c61: Status 404 returned error can't find the container with id d9bd92bb60e2b204b82aca0493cafb79d82da5c05b00a0800da6974314268c61
Jul 16 22:31:45.863662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:45.863630    2581 generic.go:356] "Generic (PLEG): container finished" podID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerID="2f0290ae1428c7af8533093a6bbdab69a5508d01cac3cf35c54fc352ef6d0ded" exitCode=6
Jul 16 22:31:45.864073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:45.863678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" event={"ID":"87a4c1ae-3507-46e2-b445-107b0bd2b0fe","Type":"ContainerDied","Data":"2f0290ae1428c7af8533093a6bbdab69a5508d01cac3cf35c54fc352ef6d0ded"}
Jul 16 22:31:45.864073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:45.863715    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" event={"ID":"87a4c1ae-3507-46e2-b445-107b0bd2b0fe","Type":"ContainerStarted","Data":"d9bd92bb60e2b204b82aca0493cafb79d82da5c05b00a0800da6974314268c61"}
Jul 16 22:31:45.864073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:45.863932    2581 scope.go:117] "RemoveContainer" containerID="2f0290ae1428c7af8533093a6bbdab69a5508d01cac3cf35c54fc352ef6d0ded"
Jul 16 22:31:46.870214 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:46.870170    2581 generic.go:356] "Generic (PLEG): container finished" podID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerID="eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b" exitCode=6
Jul 16 22:31:46.870714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:46.870262    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" event={"ID":"87a4c1ae-3507-46e2-b445-107b0bd2b0fe","Type":"ContainerDied","Data":"eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b"}
Jul 16 22:31:46.870714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:46.870339    2581 scope.go:117] "RemoveContainer" containerID="2f0290ae1428c7af8533093a6bbdab69a5508d01cac3cf35c54fc352ef6d0ded"
Jul 16 22:31:46.870714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:46.870548    2581 scope.go:117] "RemoveContainer" containerID="eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b"
Jul 16 22:31:46.870823 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:46.870785    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb_odh-ai-gateway-infra(87a4c1ae-3507-46e2-b445-107b0bd2b0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe"
Jul 16 22:31:47.876640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:47.876616    2581 scope.go:117] "RemoveContainer" containerID="eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b"
Jul 16 22:31:47.876994 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:47.876812    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb_odh-ai-gateway-infra(87a4c1ae-3507-46e2-b445-107b0bd2b0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe"
Jul 16 22:31:49.155494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:49.155458    2581 scope.go:117] "RemoveContainer" containerID="08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff"
Jul 16 22:31:49.892840 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:49.892796    2581 generic.go:356] "Generic (PLEG): container finished" podID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerID="08954c945ce33e3450dc13ad6434cba14f67f5d41f3fe82bbd0b7966f16049a4" exitCode=6
Jul 16 22:31:49.893034 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:49.892864    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerDied","Data":"08954c945ce33e3450dc13ad6434cba14f67f5d41f3fe82bbd0b7966f16049a4"}
Jul 16 22:31:49.893186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:49.893167    2581 scope.go:117] "RemoveContainer" containerID="08bbc9651ad8b7d70196bc17f3935b89c3494e78c8f0f74b7be946666541fcff"
Jul 16 22:31:49.893618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:49.893589    2581 scope.go:117] "RemoveContainer" containerID="08954c945ce33e3450dc13ad6434cba14f67f5d41f3fe82bbd0b7966f16049a4"
Jul 16 22:31:49.893824 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:49.893799    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-7pkq6_odh-ai-gateway-infra(83eb70f7-be19-43f0-b749-f574218ef9ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed"
Jul 16 22:31:50.937867 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:50.937829    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:51.077051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.077018    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:51.159068 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.159037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca\") pod \"83eb70f7-be19-43f0-b749-f574218ef9ed\" (UID: \"83eb70f7-be19-43f0-b749-f574218ef9ed\") "
Jul 16 22:31:51.159548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.159520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83eb70f7-be19-43f0-b749-f574218ef9ed" (UID: "83eb70f7-be19-43f0-b749-f574218ef9ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:51.260174 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.260068    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83eb70f7-be19-43f0-b749-f574218ef9ed-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:31:51.904913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.904883    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"
Jul 16 22:31:51.904913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.904905    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6" event={"ID":"83eb70f7-be19-43f0-b749-f574218ef9ed","Type":"ContainerDied","Data":"ba382856e378f397829c793713869668ed554e2ac66ca0f103f24c256874a2c1"}
Jul 16 22:31:51.905152 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.904971    2581 scope.go:117] "RemoveContainer" containerID="08954c945ce33e3450dc13ad6434cba14f67f5d41f3fe82bbd0b7966f16049a4"
Jul 16 22:31:51.931757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.931697    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:51.935338 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:51.935305    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-7pkq6"]
Jul 16 22:31:53.160824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:53.160793    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" path="/var/lib/kubelet/pods/83eb70f7-be19-43f0-b749-f574218ef9ed/volumes"
Jul 16 22:31:54.486460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:54.486435    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:31:54.516926 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:54.516902    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:31:55.366725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.366689    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:31:55.367053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367041    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367055    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367063    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367069    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367083    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367133    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367140    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.367293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.367145    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83eb70f7-be19-43f0-b749-f574218ef9ed" containerName="revoke-keys"
Jul 16 22:31:55.371938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.371919    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:31:55.392072 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.392042    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:31:55.494547 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.494504    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275\" (UID: \"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:31:55.595222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.595187    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275\" (UID: \"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:31:55.595822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.595803    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275\" (UID: \"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:31:55.681802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:55.681706    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:31:56.040056 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:56.040026    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:31:56.042054 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:31:56.042029    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5bdcf91c_f13d_4efe_a3b1_0d7e159e52e8.slice/crio-fb15490004d1dab991258b5c9bd922bc54e9a35d5680809e6603009708b7c1e9 WatchSource:0}: Error finding container fb15490004d1dab991258b5c9bd922bc54e9a35d5680809e6603009708b7c1e9: Status 404 returned error can't find the container with id fb15490004d1dab991258b5c9bd922bc54e9a35d5680809e6603009708b7c1e9
Jul 16 22:31:56.929749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:56.929661    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerID="8fb5cc9bc966688bf28f427267313c8cff0b3d7627f3d6beabbd8422ad28d872" exitCode=6
Jul 16 22:31:56.930106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:56.929745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" event={"ID":"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8","Type":"ContainerDied","Data":"8fb5cc9bc966688bf28f427267313c8cff0b3d7627f3d6beabbd8422ad28d872"}
Jul 16 22:31:56.930106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:56.929800    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" event={"ID":"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8","Type":"ContainerStarted","Data":"fb15490004d1dab991258b5c9bd922bc54e9a35d5680809e6603009708b7c1e9"}
Jul 16 22:31:56.930106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:56.929922    2581 scope.go:117] "RemoveContainer" containerID="8fb5cc9bc966688bf28f427267313c8cff0b3d7627f3d6beabbd8422ad28d872"
Jul 16 22:31:57.936373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:57.936284    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerID="def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1" exitCode=6
Jul 16 22:31:57.936908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:57.936367    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" event={"ID":"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8","Type":"ContainerDied","Data":"def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1"}
Jul 16 22:31:57.936908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:57.936425    2581 scope.go:117] "RemoveContainer" containerID="8fb5cc9bc966688bf28f427267313c8cff0b3d7627f3d6beabbd8422ad28d872"
Jul 16 22:31:57.936908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:57.936675    2581 scope.go:117] "RemoveContainer" containerID="def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1"
Jul 16 22:31:57.936908 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:57.936892    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275_odh-ai-gateway-infra(5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8"
Jul 16 22:31:58.942520 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:31:58.942485    2581 scope.go:117] "RemoveContainer" containerID="def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1"
Jul 16 22:31:58.942914 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:31:58.942654    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275_odh-ai-gateway-infra(5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8"
Jul 16 22:32:03.156449 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:03.156413    2581 scope.go:117] "RemoveContainer" containerID="eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b"
Jul 16 22:32:03.965752 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:03.965717    2581 generic.go:356] "Generic (PLEG): container finished" podID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerID="b5276e33157e7b9df787eda1e924510e660cea868aedc08e446f86ee9e8da880" exitCode=6
Jul 16 22:32:03.965956 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:03.965784    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" event={"ID":"87a4c1ae-3507-46e2-b445-107b0bd2b0fe","Type":"ContainerDied","Data":"b5276e33157e7b9df787eda1e924510e660cea868aedc08e446f86ee9e8da880"}
Jul 16 22:32:03.965956 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:03.965845    2581 scope.go:117] "RemoveContainer" containerID="eb8460d2a8d8624fcc67f241e649fcba3ff26d5f6745305795480457f2318e4b"
Jul 16 22:32:03.966211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:03.966179    2581 scope.go:117] "RemoveContainer" containerID="b5276e33157e7b9df787eda1e924510e660cea868aedc08e446f86ee9e8da880"
Jul 16 22:32:03.966404 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:03.966386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb_odh-ai-gateway-infra(87a4c1ae-3507-46e2-b445-107b0bd2b0fe)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe"
Jul 16 22:32:05.018580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.018542    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:32:05.144223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.144198    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:32:05.282601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.282502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca\") pod \"87a4c1ae-3507-46e2-b445-107b0bd2b0fe\" (UID: \"87a4c1ae-3507-46e2-b445-107b0bd2b0fe\") "
Jul 16 22:32:05.282943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.282912    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "87a4c1ae-3507-46e2-b445-107b0bd2b0fe" (UID: "87a4c1ae-3507-46e2-b445-107b0bd2b0fe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:05.383714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.383674    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/87a4c1ae-3507-46e2-b445-107b0bd2b0fe-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:32:05.977418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.977387    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"
Jul 16 22:32:05.977609 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.977385    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb" event={"ID":"87a4c1ae-3507-46e2-b445-107b0bd2b0fe","Type":"ContainerDied","Data":"d9bd92bb60e2b204b82aca0493cafb79d82da5c05b00a0800da6974314268c61"}
Jul 16 22:32:05.977609 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:05.977527    2581 scope.go:117] "RemoveContainer" containerID="b5276e33157e7b9df787eda1e924510e660cea868aedc08e446f86ee9e8da880"
Jul 16 22:32:06.004807 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.004779    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:32:06.009629 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.009593    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xh6xb"]
Jul 16 22:32:06.918086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918051    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918380    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918392    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918405    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918411    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918420    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918427    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918481    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.918505 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.918489    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:06.922400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.922381    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:06.936372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:06.936345    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:07.096829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.096794    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642\" (UID: \"eb72f46a-3cc2-44b7-bd40-1904441be950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:07.160821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.160791    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" path="/var/lib/kubelet/pods/87a4c1ae-3507-46e2-b445-107b0bd2b0fe/volumes"
Jul 16 22:32:07.197830 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.197739    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642\" (UID: \"eb72f46a-3cc2-44b7-bd40-1904441be950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:07.198384 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.198365    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642\" (UID: \"eb72f46a-3cc2-44b7-bd40-1904441be950\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:07.231992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.231962    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:07.362080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.362052    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:07.363998 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:32:07.363968    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb72f46a_3cc2_44b7_bd40_1904441be950.slice/crio-e97d8a1da83fdf0a496dc98174818cfccbb37a2c05b84dbdb51ad22de7dd9086 WatchSource:0}: Error finding container e97d8a1da83fdf0a496dc98174818cfccbb37a2c05b84dbdb51ad22de7dd9086: Status 404 returned error can't find the container with id e97d8a1da83fdf0a496dc98174818cfccbb37a2c05b84dbdb51ad22de7dd9086
Jul 16 22:32:07.988139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.988099    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerID="05e58d373e484fe71cfdd3b0ab8fcbe6a5d1092a0a5eba5804a2d773f55880aa" exitCode=6
Jul 16 22:32:07.988493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.988155    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" event={"ID":"eb72f46a-3cc2-44b7-bd40-1904441be950","Type":"ContainerDied","Data":"05e58d373e484fe71cfdd3b0ab8fcbe6a5d1092a0a5eba5804a2d773f55880aa"}
Jul 16 22:32:07.988493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.988199    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" event={"ID":"eb72f46a-3cc2-44b7-bd40-1904441be950","Type":"ContainerStarted","Data":"e97d8a1da83fdf0a496dc98174818cfccbb37a2c05b84dbdb51ad22de7dd9086"}
Jul 16 22:32:07.988493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:07.988421    2581 scope.go:117] "RemoveContainer" containerID="05e58d373e484fe71cfdd3b0ab8fcbe6a5d1092a0a5eba5804a2d773f55880aa"
Jul 16 22:32:08.995000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:08.994904    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerID="65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5" exitCode=6
Jul 16 22:32:08.995000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:08.994988    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" event={"ID":"eb72f46a-3cc2-44b7-bd40-1904441be950","Type":"ContainerDied","Data":"65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5"}
Jul 16 22:32:08.995503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:08.995049    2581 scope.go:117] "RemoveContainer" containerID="05e58d373e484fe71cfdd3b0ab8fcbe6a5d1092a0a5eba5804a2d773f55880aa"
Jul 16 22:32:08.995503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:08.995214    2581 scope.go:117] "RemoveContainer" containerID="65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5"
Jul 16 22:32:08.995503 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:08.995466    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642_odh-ai-gateway-infra(eb72f46a-3cc2-44b7-bd40-1904441be950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950"
Jul 16 22:32:10.007099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:10.007070    2581 scope.go:117] "RemoveContainer" containerID="65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5"
Jul 16 22:32:10.007501 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:10.007267    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642_odh-ai-gateway-infra(eb72f46a-3cc2-44b7-bd40-1904441be950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950"
Jul 16 22:32:12.155577 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:12.155544    2581 scope.go:117] "RemoveContainer" containerID="def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1"
Jul 16 22:32:13.021320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:13.021219    2581 generic.go:356] "Generic (PLEG): container finished" podID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerID="a71247bd00b63dfd246ca0545b38c7388e9151531672a9fafb909eb6f2f44e76" exitCode=6
Jul 16 22:32:13.021320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:13.021297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" event={"ID":"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8","Type":"ContainerDied","Data":"a71247bd00b63dfd246ca0545b38c7388e9151531672a9fafb909eb6f2f44e76"}
Jul 16 22:32:13.021530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:13.021363    2581 scope.go:117] "RemoveContainer" containerID="def0f3ef82c621d34eef90338c8f1aca16b9f7084714efa9b792ef00d858d5b1"
Jul 16 22:32:13.021696 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:13.021683    2581 scope.go:117] "RemoveContainer" containerID="a71247bd00b63dfd246ca0545b38c7388e9151531672a9fafb909eb6f2f44e76"
Jul 16 22:32:13.021926 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:13.021903    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275_odh-ai-gateway-infra(5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8"
Jul 16 22:32:14.079053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:14.079020    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:32:14.218548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:14.218514    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:32:14.361648 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:14.361620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca\") pod \"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8\" (UID: \"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8\") "
Jul 16 22:32:14.362006 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:14.361981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" (UID: "5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:14.462248 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:14.462207    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:32:15.033082 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.033052    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"
Jul 16 22:32:15.033328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.033085    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275" event={"ID":"5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8","Type":"ContainerDied","Data":"fb15490004d1dab991258b5c9bd922bc54e9a35d5680809e6603009708b7c1e9"}
Jul 16 22:32:15.033328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.033144    2581 scope.go:117] "RemoveContainer" containerID="a71247bd00b63dfd246ca0545b38c7388e9151531672a9fafb909eb6f2f44e76"
Jul 16 22:32:15.061711 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.061678    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:32:15.069991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.069963    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-bv275"]
Jul 16 22:32:15.160273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:15.160218    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" path="/var/lib/kubelet/pods/5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8/volumes"
Jul 16 22:32:23.009014 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.008979    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009342    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009366    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009371    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009376    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009453    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009462    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:23.009658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.009470    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="87a4c1ae-3507-46e2-b445-107b0bd2b0fe" containerName="revoke-keys"
Jul 16 22:32:23.013436 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.013418    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:23.021749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.021717    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:23.023329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.023305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qxp2n\" (UID: \"57a67f57-84a7-461b-85a9-4a6c58cdfc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:23.124455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.124429    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qxp2n\" (UID: \"57a67f57-84a7-461b-85a9-4a6c58cdfc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:23.125028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.125009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-qxp2n\" (UID: \"57a67f57-84a7-461b-85a9-4a6c58cdfc7e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:23.324811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.324728    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:23.453421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:23.453393    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:23.455379 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:32:23.455343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57a67f57_84a7_461b_85a9_4a6c58cdfc7e.slice/crio-6d586dab7bb0f9205c3ccdae43a23a0e13b52d6e45bbdfee025ae6cb98d8a2eb WatchSource:0}: Error finding container 6d586dab7bb0f9205c3ccdae43a23a0e13b52d6e45bbdfee025ae6cb98d8a2eb: Status 404 returned error can't find the container with id 6d586dab7bb0f9205c3ccdae43a23a0e13b52d6e45bbdfee025ae6cb98d8a2eb
Jul 16 22:32:24.073344 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:24.073315    2581 generic.go:356] "Generic (PLEG): container finished" podID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerID="eca7429239f8f3ea752776f63e2bbf10ece60972df5254e7aa93f96c10cf298e" exitCode=6
Jul 16 22:32:24.073707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:24.073387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" event={"ID":"57a67f57-84a7-461b-85a9-4a6c58cdfc7e","Type":"ContainerDied","Data":"eca7429239f8f3ea752776f63e2bbf10ece60972df5254e7aa93f96c10cf298e"}
Jul 16 22:32:24.073707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:24.073434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" event={"ID":"57a67f57-84a7-461b-85a9-4a6c58cdfc7e","Type":"ContainerStarted","Data":"6d586dab7bb0f9205c3ccdae43a23a0e13b52d6e45bbdfee025ae6cb98d8a2eb"}
Jul 16 22:32:24.073707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:24.073611    2581 scope.go:117] "RemoveContainer" containerID="eca7429239f8f3ea752776f63e2bbf10ece60972df5254e7aa93f96c10cf298e"
Jul 16 22:32:24.155448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:24.155424    2581 scope.go:117] "RemoveContainer" containerID="65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5"
Jul 16 22:32:25.079421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.079326    2581 generic.go:356] "Generic (PLEG): container finished" podID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerID="810251a061e385b1761d7f45c63285848d1aabf74d686f86d8b22983dd3c0c82" exitCode=6
Jul 16 22:32:25.079421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.079395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" event={"ID":"eb72f46a-3cc2-44b7-bd40-1904441be950","Type":"ContainerDied","Data":"810251a061e385b1761d7f45c63285848d1aabf74d686f86d8b22983dd3c0c82"}
Jul 16 22:32:25.079919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.079487    2581 scope.go:117] "RemoveContainer" containerID="65e3ca6770366d86b70417e3bfd7129f18d6fc211f63a252b6bdb3ccd98feae5"
Jul 16 22:32:25.079919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.079796    2581 scope.go:117] "RemoveContainer" containerID="810251a061e385b1761d7f45c63285848d1aabf74d686f86d8b22983dd3c0c82"
Jul 16 22:32:25.080039 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:25.080016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642_odh-ai-gateway-infra(eb72f46a-3cc2-44b7-bd40-1904441be950)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950"
Jul 16 22:32:25.081398 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.081373    2581 generic.go:356] "Generic (PLEG): container finished" podID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerID="149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5" exitCode=6
Jul 16 22:32:25.081501 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.081452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" event={"ID":"57a67f57-84a7-461b-85a9-4a6c58cdfc7e","Type":"ContainerDied","Data":"149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5"}
Jul 16 22:32:25.081667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.081655    2581 scope.go:117] "RemoveContainer" containerID="149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5"
Jul 16 22:32:25.081845 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:25.081829    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qxp2n_odh-ai-gateway-infra(57a67f57-84a7-461b-85a9-4a6c58cdfc7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e"
Jul 16 22:32:25.090178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:25.090164    2581 scope.go:117] "RemoveContainer" containerID="eca7429239f8f3ea752776f63e2bbf10ece60972df5254e7aa93f96c10cf298e"
Jul 16 22:32:26.088628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.088603    2581 scope.go:117] "RemoveContainer" containerID="149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5"
Jul 16 22:32:26.089005 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:26.088794    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qxp2n_odh-ai-gateway-infra(57a67f57-84a7-461b-85a9-4a6c58cdfc7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e"
Jul 16 22:32:26.120459 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.120434    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:26.255677 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.255646    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:26.350273 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.350174    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca\") pod \"eb72f46a-3cc2-44b7-bd40-1904441be950\" (UID: \"eb72f46a-3cc2-44b7-bd40-1904441be950\") "
Jul 16 22:32:26.350543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.350520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eb72f46a-3cc2-44b7-bd40-1904441be950" (UID: "eb72f46a-3cc2-44b7-bd40-1904441be950"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:26.450886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:26.450861    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eb72f46a-3cc2-44b7-bd40-1904441be950-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:32:27.094239 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.094205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642" event={"ID":"eb72f46a-3cc2-44b7-bd40-1904441be950","Type":"ContainerDied","Data":"e97d8a1da83fdf0a496dc98174818cfccbb37a2c05b84dbdb51ad22de7dd9086"}
Jul 16 22:32:27.094699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.094287    2581 scope.go:117] "RemoveContainer" containerID="810251a061e385b1761d7f45c63285848d1aabf74d686f86d8b22983dd3c0c82"
Jul 16 22:32:27.094699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.094212    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"
Jul 16 22:32:27.116000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.115971    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:27.119985 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.119962    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-4m642"]
Jul 16 22:32:27.161033 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:27.161004    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" path="/var/lib/kubelet/pods/eb72f46a-3cc2-44b7-bd40-1904441be950/volumes"
Jul 16 22:32:40.154703 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:40.154672    2581 scope.go:117] "RemoveContainer" containerID="149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5"
Jul 16 22:32:41.156417 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:41.156382    2581 generic.go:356] "Generic (PLEG): container finished" podID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerID="4a97db0ed19619feed08a45e6e92701a10e63201b958e0cade58f3b3a910495c" exitCode=6
Jul 16 22:32:41.159816 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:41.159789    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" event={"ID":"57a67f57-84a7-461b-85a9-4a6c58cdfc7e","Type":"ContainerDied","Data":"4a97db0ed19619feed08a45e6e92701a10e63201b958e0cade58f3b3a910495c"}
Jul 16 22:32:41.159941 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:41.159849    2581 scope.go:117] "RemoveContainer" containerID="149813835eb06ddc03bdd28a126f58d77559e1a58d4abb191e9a28e3888de1e5"
Jul 16 22:32:41.160281 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:41.160264    2581 scope.go:117] "RemoveContainer" containerID="4a97db0ed19619feed08a45e6e92701a10e63201b958e0cade58f3b3a910495c"
Jul 16 22:32:41.160474 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:41.160453    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-qxp2n_odh-ai-gateway-infra(57a67f57-84a7-461b-85a9-4a6c58cdfc7e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e"
Jul 16 22:32:42.193242 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.193205    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:42.227815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.227783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:32:42.325928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.325906    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:42.377458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.377430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca\") pod \"57a67f57-84a7-461b-85a9-4a6c58cdfc7e\" (UID: \"57a67f57-84a7-461b-85a9-4a6c58cdfc7e\") "
Jul 16 22:32:42.377805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.377780    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "57a67f57-84a7-461b-85a9-4a6c58cdfc7e" (UID: "57a67f57-84a7-461b-85a9-4a6c58cdfc7e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:42.478648 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:42.478569    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/57a67f57-84a7-461b-85a9-4a6c58cdfc7e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:32:43.168507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:43.168482    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"
Jul 16 22:32:43.168507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:43.168496    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n" event={"ID":"57a67f57-84a7-461b-85a9-4a6c58cdfc7e","Type":"ContainerDied","Data":"6d586dab7bb0f9205c3ccdae43a23a0e13b52d6e45bbdfee025ae6cb98d8a2eb"}
Jul 16 22:32:43.168704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:43.168557    2581 scope.go:117] "RemoveContainer" containerID="4a97db0ed19619feed08a45e6e92701a10e63201b958e0cade58f3b3a910495c"
Jul 16 22:32:43.185537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:43.185507    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:43.190530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:43.190503    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-qxp2n"]
Jul 16 22:32:44.536643 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.536600    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537046    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537065    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537078    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537087    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537100    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537108    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537126    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537162    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537253    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537268    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537278    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537288    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:44.537494 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.537296    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5bdcf91c-f13d-4efe-a3b1-0d7e159e52e8" containerName="revoke-keys"
Jul 16 22:32:44.541619 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.541595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:32:44.545576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.545524    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:32:44.546346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.546314    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:32:44.547788 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.547765    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:32:44.598651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.598611    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w\" (UID: \"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:32:44.699024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.698968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w\" (UID: \"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:32:44.699653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.699629    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w\" (UID: \"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:32:44.853012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.852982    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:32:44.983884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:44.983855    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:32:44.986372 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:32:44.986343    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28e2c40e_3b88_4bd8_bde6_70d734c2b8d7.slice/crio-7aa8e38fc504b5718ad7a7621c29c2d1144cb223fb76cfa05d8f8accd594d22e WatchSource:0}: Error finding container 7aa8e38fc504b5718ad7a7621c29c2d1144cb223fb76cfa05d8f8accd594d22e: Status 404 returned error can't find the container with id 7aa8e38fc504b5718ad7a7621c29c2d1144cb223fb76cfa05d8f8accd594d22e
Jul 16 22:32:45.160400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.160323    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" path="/var/lib/kubelet/pods/57a67f57-84a7-461b-85a9-4a6c58cdfc7e/volumes"
Jul 16 22:32:45.179353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.179317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" event={"ID":"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7","Type":"ContainerStarted","Data":"7aa8e38fc504b5718ad7a7621c29c2d1144cb223fb76cfa05d8f8accd594d22e"}
Jul 16 22:32:45.384480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.384446    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:32:45.384784 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.384771    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:45.384826 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.384786    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:45.384862 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.384850    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="57a67f57-84a7-461b-85a9-4a6c58cdfc7e" containerName="revoke-keys"
Jul 16 22:32:45.384895 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.384861    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb72f46a-3cc2-44b7-bd40-1904441be950" containerName="revoke-keys"
Jul 16 22:32:45.387821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.387805    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:32:45.395491 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.395462    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:32:45.505690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.505617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x\" (UID: \"6175df36-4fdd-4495-86fe-39d3c93afc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:32:45.607088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.607058    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x\" (UID: \"6175df36-4fdd-4495-86fe-39d3c93afc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:32:45.607684 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.607665    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x\" (UID: \"6175df36-4fdd-4495-86fe-39d3c93afc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:32:45.699289 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.699247    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:32:45.827530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:45.827353    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:32:45.829969 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:32:45.829939    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6175df36_4fdd_4495_86fe_39d3c93afc00.slice/crio-a0f46ba71781f000c69589c04d07370440c92f17f6d7af58bdac09f84be6d965 WatchSource:0}: Error finding container a0f46ba71781f000c69589c04d07370440c92f17f6d7af58bdac09f84be6d965: Status 404 returned error can't find the container with id a0f46ba71781f000c69589c04d07370440c92f17f6d7af58bdac09f84be6d965
Jul 16 22:32:46.184848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.184815    2581 generic.go:356] "Generic (PLEG): container finished" podID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerID="7a402d59a79fe5009d736d9e49d36ff5aa159515dc5c8615908dd9fc67f600a3" exitCode=6
Jul 16 22:32:46.185018 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.184909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" event={"ID":"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7","Type":"ContainerDied","Data":"7a402d59a79fe5009d736d9e49d36ff5aa159515dc5c8615908dd9fc67f600a3"}
Jul 16 22:32:46.185126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.185106    2581 scope.go:117] "RemoveContainer" containerID="7a402d59a79fe5009d736d9e49d36ff5aa159515dc5c8615908dd9fc67f600a3"
Jul 16 22:32:46.186666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.186641    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerStarted","Data":"00538a31223810e4c48101f37d8c04cd662b4d4ba3b964b8a69c0cd72876fa15"}
Jul 16 22:32:46.186785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.186695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerStarted","Data":"a0f46ba71781f000c69589c04d07370440c92f17f6d7af58bdac09f84be6d965"}
Jul 16 22:32:46.217155 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:46.217103    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" podStartSLOduration=1.217087555 podStartE2EDuration="1.217087555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:32:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:46.216941208 +0000 UTC m=+2753.546381123" watchObservedRunningTime="2026-07-16 22:32:46.217087555 +0000 UTC m=+2753.546527458"
Jul 16 22:32:47.192512 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.192421    2581 generic.go:356] "Generic (PLEG): container finished" podID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerID="112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e" exitCode=6
Jul 16 22:32:47.192941 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.192507    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" event={"ID":"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7","Type":"ContainerDied","Data":"112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e"}
Jul 16 22:32:47.192941 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.192574    2581 scope.go:117] "RemoveContainer" containerID="7a402d59a79fe5009d736d9e49d36ff5aa159515dc5c8615908dd9fc67f600a3"
Jul 16 22:32:47.192941 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.192759    2581 scope.go:117] "RemoveContainer" containerID="112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e"
Jul 16 22:32:47.193094 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:47.192980    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w_odh-ai-gateway-infra(28e2c40e-3b88-4bd8-bde6-70d734c2b8d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7"
Jul 16 22:32:47.194047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.194024    2581 generic.go:356] "Generic (PLEG): container finished" podID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerID="00538a31223810e4c48101f37d8c04cd662b4d4ba3b964b8a69c0cd72876fa15" exitCode=6
Jul 16 22:32:47.194126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.194103    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerDied","Data":"00538a31223810e4c48101f37d8c04cd662b4d4ba3b964b8a69c0cd72876fa15"}
Jul 16 22:32:47.194278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:47.194265    2581 scope.go:117] "RemoveContainer" containerID="00538a31223810e4c48101f37d8c04cd662b4d4ba3b964b8a69c0cd72876fa15"
Jul 16 22:32:48.200457 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:48.200414    2581 scope.go:117] "RemoveContainer" containerID="112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e"
Jul 16 22:32:48.200873 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:48.200676    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w_odh-ai-gateway-infra(28e2c40e-3b88-4bd8-bde6-70d734c2b8d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7"
Jul 16 22:32:48.202017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:48.201993    2581 generic.go:356] "Generic (PLEG): container finished" podID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerID="81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00" exitCode=6
Jul 16 22:32:48.202116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:48.202072    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerDied","Data":"81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00"}
Jul 16 22:32:48.202154 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:48.202129    2581 scope.go:117] "RemoveContainer" containerID="00538a31223810e4c48101f37d8c04cd662b4d4ba3b964b8a69c0cd72876fa15"
Jul 16 22:32:48.202351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:48.202336    2581 scope.go:117] "RemoveContainer" containerID="81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00"
Jul 16 22:32:48.202543 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:48.202526    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x_odh-ai-gateway-infra(6175df36-4fdd-4495-86fe-39d3c93afc00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00"
Jul 16 22:32:49.207427 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:49.207403    2581 scope.go:117] "RemoveContainer" containerID="81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00"
Jul 16 22:32:49.207790 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:32:49.207590    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x_odh-ai-gateway-infra(6175df36-4fdd-4495-86fe-39d3c93afc00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00"
Jul 16 22:32:50.325335 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:50.325300    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:32:57.056122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.056083    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:32:57.061080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.061060    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.065509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.065491    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-50ab3915-openshift-default-dockercfg-ljczl\""
Jul 16 22:32:57.078182 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.078151    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:32:57.213024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.212987    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhxpp\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213052    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213192    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213273    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213298    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213320    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.213379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.213366    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.314914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.314804    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.314914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.314844    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhxpp\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.314914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.314862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.314914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.314904    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315074    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315131    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315161    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315201    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315257    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315283    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315425    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315545    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315555    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.315727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.315671    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.317814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.317791    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.318372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.318350    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.324060 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.324031    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.324264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.324222    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhxpp\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp\") pod \"e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") " pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.371250 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.371193    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:57.510559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.510531    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:32:57.512091 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:32:57.512066    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a5831ec_2a59_4294_ba9f_8031da33d8cd.slice/crio-ccbf52bdd225099f521419eb8f4073a54eb1a0553ed9e4e875e5263a70e2723a WatchSource:0}: Error finding container ccbf52bdd225099f521419eb8f4073a54eb1a0553ed9e4e875e5263a70e2723a: Status 404 returned error can't find the container with id ccbf52bdd225099f521419eb8f4073a54eb1a0553ed9e4e875e5263a70e2723a
Jul 16 22:32:57.514367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.514331    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:32:57.514484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.514399    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:32:57.514484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:57.514433    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:32:58.253864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:58.253825    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerStarted","Data":"e8e8a65403861eaf532f2ed908a1a295a171b2816e4cd008ee37108814a195f2"}
Jul 16 22:32:58.254290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:58.253912    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerStarted","Data":"ccbf52bdd225099f521419eb8f4073a54eb1a0553ed9e4e875e5263a70e2723a"}
Jul 16 22:32:58.281403 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:58.281352    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podStartSLOduration=1.28133659 podStartE2EDuration="1.28133659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:32:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:32:58.280638901 +0000 UTC m=+2765.610078805" watchObservedRunningTime="2026-07-16 22:32:58.28133659 +0000 UTC m=+2765.610776493"
Jul 16 22:32:58.371679 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:58.371643    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:58.376482 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:58.376458    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:59.216002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:59.215962    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:32:59.258105 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:59.258067    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:32:59.259080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:32:59.259061    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:01.155522 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:01.155493    2581 scope.go:117] "RemoveContainer" containerID="81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00"
Jul 16 22:33:02.272895 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:02.272863    2581 generic.go:356] "Generic (PLEG): container finished" podID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerID="39f2490dc65b290a91bf7ffc856f829798b5b15f56119a99f4b9ea444e5ab1e7" exitCode=6
Jul 16 22:33:02.273345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:02.272936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerDied","Data":"39f2490dc65b290a91bf7ffc856f829798b5b15f56119a99f4b9ea444e5ab1e7"}
Jul 16 22:33:02.273345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:02.272995    2581 scope.go:117] "RemoveContainer" containerID="81d2a8a72d1c16f2fb8a6b780532bdaeec3665ed1fa970db8f3808249f17ea00"
Jul 16 22:33:02.273345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:02.273296    2581 scope.go:117] "RemoveContainer" containerID="39f2490dc65b290a91bf7ffc856f829798b5b15f56119a99f4b9ea444e5ab1e7"
Jul 16 22:33:02.273500 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:02.273477    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x_odh-ai-gateway-infra(6175df36-4fdd-4495-86fe-39d3c93afc00)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00"
Jul 16 22:33:03.157160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.157129    2581 scope.go:117] "RemoveContainer" containerID="112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e"
Jul 16 22:33:03.306962 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.306934    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:33:03.440388 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.440363    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:33:03.573063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.573032    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca\") pod \"6175df36-4fdd-4495-86fe-39d3c93afc00\" (UID: \"6175df36-4fdd-4495-86fe-39d3c93afc00\") "
Jul 16 22:33:03.573434 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.573416    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6175df36-4fdd-4495-86fe-39d3c93afc00" (UID: "6175df36-4fdd-4495-86fe-39d3c93afc00"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:03.673989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:03.673910    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6175df36-4fdd-4495-86fe-39d3c93afc00-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:04.287435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.287401    2581 generic.go:356] "Generic (PLEG): container finished" podID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerID="649ba924a6ab8f129a7261c761286f503d7515b1f84a69dd3c1bbab0834bce77" exitCode=6
Jul 16 22:33:04.287624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.287477    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" event={"ID":"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7","Type":"ContainerDied","Data":"649ba924a6ab8f129a7261c761286f503d7515b1f84a69dd3c1bbab0834bce77"}
Jul 16 22:33:04.287624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.287554    2581 scope.go:117] "RemoveContainer" containerID="112892f747b6cab08708d145ddac40d76c991918efdff33f1bcdce1a19bcb52e"
Jul 16 22:33:04.287903 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.287884    2581 scope.go:117] "RemoveContainer" containerID="649ba924a6ab8f129a7261c761286f503d7515b1f84a69dd3c1bbab0834bce77"
Jul 16 22:33:04.288147 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:04.288110    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w_odh-ai-gateway-infra(28e2c40e-3b88-4bd8-bde6-70d734c2b8d7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7"
Jul 16 22:33:04.288922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.288905    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"
Jul 16 22:33:04.288922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.288917    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x" event={"ID":"6175df36-4fdd-4495-86fe-39d3c93afc00","Type":"ContainerDied","Data":"a0f46ba71781f000c69589c04d07370440c92f17f6d7af58bdac09f84be6d965"}
Jul 16 22:33:04.298813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.298796    2581 scope.go:117] "RemoveContainer" containerID="39f2490dc65b290a91bf7ffc856f829798b5b15f56119a99f4b9ea444e5ab1e7"
Jul 16 22:33:04.321885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.321864    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:33:04.327573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:04.327552    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-wrg8x"]
Jul 16 22:33:05.160260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.160208    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" path="/var/lib/kubelet/pods/6175df36-4fdd-4495-86fe-39d3c93afc00/volumes"
Jul 16 22:33:05.318142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.318111    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:33:05.451204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.451174    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:33:05.590122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.590090    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca\") pod \"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7\" (UID: \"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7\") "
Jul 16 22:33:05.590549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.590520    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" (UID: "28e2c40e-3b88-4bd8-bde6-70d734c2b8d7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:05.691452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:05.691377    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:06.302329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.302302    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"
Jul 16 22:33:06.302497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.302327    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w" event={"ID":"28e2c40e-3b88-4bd8-bde6-70d734c2b8d7","Type":"ContainerDied","Data":"7aa8e38fc504b5718ad7a7621c29c2d1144cb223fb76cfa05d8f8accd594d22e"}
Jul 16 22:33:06.302497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.302392    2581 scope.go:117] "RemoveContainer" containerID="649ba924a6ab8f129a7261c761286f503d7515b1f84a69dd3c1bbab0834bce77"
Jul 16 22:33:06.323691 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.323661    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:33:06.329884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.329863    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-pq96w"]
Jul 16 22:33:06.957327 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.957292    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:06.958379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958352    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.958540 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958526    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.958627 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958617    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.958696 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958687    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.958775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958765    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.958849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958840    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.958928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958917    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959005 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.958995    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959081    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959180 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959169    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959364    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959460    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.959555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959546    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.959627 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959619    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:06.959695 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959686    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.959772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.959762    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6175df36-4fdd-4495-86fe-39d3c93afc00" containerName="revoke-keys"
Jul 16 22:33:06.964794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.964776    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:06.967785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.967765    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:06.968160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.968142    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:33:06.972091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:06.972069    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:07.106195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.106162    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b\" (UID: \"d8aa11c3-7920-46cf-af51-ae259b43314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:07.164615 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.164572    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" path="/var/lib/kubelet/pods/28e2c40e-3b88-4bd8-bde6-70d734c2b8d7/volumes"
Jul 16 22:33:07.207005 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.206974    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b\" (UID: \"d8aa11c3-7920-46cf-af51-ae259b43314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:07.207626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.207583    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b\" (UID: \"d8aa11c3-7920-46cf-af51-ae259b43314f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:07.279063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.279026    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:07.420018 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:07.419995    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:07.421997 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:07.421970    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8aa11c3_7920_46cf_af51_ae259b43314f.slice/crio-ef241c4413e466a33c5c4e644f7dc3b23ee633143cad0851961300faa98d3cfc WatchSource:0}: Error finding container ef241c4413e466a33c5c4e644f7dc3b23ee633143cad0851961300faa98d3cfc: Status 404 returned error can't find the container with id ef241c4413e466a33c5c4e644f7dc3b23ee633143cad0851961300faa98d3cfc
Jul 16 22:33:08.313779 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:08.313751    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerID="9e10e18bc2300c7d55dc0215a35cd7610f0d853b6793e38587ea4aa4bc63ec06" exitCode=6
Jul 16 22:33:08.314125 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:08.313839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" event={"ID":"d8aa11c3-7920-46cf-af51-ae259b43314f","Type":"ContainerDied","Data":"9e10e18bc2300c7d55dc0215a35cd7610f0d853b6793e38587ea4aa4bc63ec06"}
Jul 16 22:33:08.314125 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:08.313910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" event={"ID":"d8aa11c3-7920-46cf-af51-ae259b43314f","Type":"ContainerStarted","Data":"ef241c4413e466a33c5c4e644f7dc3b23ee633143cad0851961300faa98d3cfc"}
Jul 16 22:33:08.314125 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:08.313964    2581 scope.go:117] "RemoveContainer" containerID="9e10e18bc2300c7d55dc0215a35cd7610f0d853b6793e38587ea4aa4bc63ec06"
Jul 16 22:33:09.320197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:09.320159    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerID="a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab" exitCode=6
Jul 16 22:33:09.320635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:09.320201    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" event={"ID":"d8aa11c3-7920-46cf-af51-ae259b43314f","Type":"ContainerDied","Data":"a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab"}
Jul 16 22:33:09.320635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:09.320272    2581 scope.go:117] "RemoveContainer" containerID="9e10e18bc2300c7d55dc0215a35cd7610f0d853b6793e38587ea4aa4bc63ec06"
Jul 16 22:33:09.320635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:09.320521    2581 scope.go:117] "RemoveContainer" containerID="a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab"
Jul 16 22:33:09.320803 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:09.320774    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b_odh-ai-gateway-infra(d8aa11c3-7920-46cf-af51-ae259b43314f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f"
Jul 16 22:33:09.617992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:09.617964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:33:10.326985 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:10.326954    2581 scope.go:117] "RemoveContainer" containerID="a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab"
Jul 16 22:33:10.327409 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:10.327205    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b_odh-ai-gateway-infra(d8aa11c3-7920-46cf-af51-ae259b43314f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f"
Jul 16 22:33:12.578358 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.578320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"]
Jul 16 22:33:12.578736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.578652    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:12.578736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.578663    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="28e2c40e-3b88-4bd8-bde6-70d734c2b8d7" containerName="revoke-keys"
Jul 16 22:33:12.582880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.582863    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:12.588949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.588915    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"]
Jul 16 22:33:12.654432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.654395    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh\" (UID: \"83e480fb-3074-42ce-9d89-9dc3fdc4af92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:12.755277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.755240    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh\" (UID: \"83e480fb-3074-42ce-9d89-9dc3fdc4af92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:12.755808 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.755783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh\" (UID: \"83e480fb-3074-42ce-9d89-9dc3fdc4af92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:12.895404 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:12.895367    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:13.048434 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.048408    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"]
Jul 16 22:33:13.050683 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:13.050660    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83e480fb_3074_42ce_9d89_9dc3fdc4af92.slice/crio-c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56 WatchSource:0}: Error finding container c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56: Status 404 returned error can't find the container with id c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56
Jul 16 22:33:13.342922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.342886    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerID="e8e8a65403861eaf532f2ed908a1a295a171b2816e4cd008ee37108814a195f2" exitCode=0
Jul 16 22:33:13.343097 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.342965    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerDied","Data":"e8e8a65403861eaf532f2ed908a1a295a171b2816e4cd008ee37108814a195f2"}
Jul 16 22:33:13.343425 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.343401    2581 scope.go:117] "RemoveContainer" containerID="e8e8a65403861eaf532f2ed908a1a295a171b2816e4cd008ee37108814a195f2"
Jul 16 22:33:13.344260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.344214    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh" event={"ID":"83e480fb-3074-42ce-9d89-9dc3fdc4af92","Type":"ContainerStarted","Data":"c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56"}
Jul 16 22:33:13.344546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.344521    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:33:13.344606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.344589    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:33:13.344679 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:13.344662    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:33:14.258451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.258335    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:14.353129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.353084    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerStarted","Data":"0db9ee7b96c92d97dd3bc11dd445cb0540f2a774717422544e4aff127483dad8"}
Jul 16 22:33:14.356267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.356211    2581 generic.go:356] "Generic (PLEG): container finished" podID="83e480fb-3074-42ce-9d89-9dc3fdc4af92" containerID="1c12d712dc0509eba1f2bf0b7f43f7c88799406a63096ad281cfeb2a47908f2d" exitCode=0
Jul 16 22:33:14.356446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.356356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh" event={"ID":"83e480fb-3074-42ce-9d89-9dc3fdc4af92","Type":"ContainerDied","Data":"1c12d712dc0509eba1f2bf0b7f43f7c88799406a63096ad281cfeb2a47908f2d"}
Jul 16 22:33:14.371545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.371505    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:14.372811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.372785    2581 patch_prober.go:28] interesting pod/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 16 22:33:14.372954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:14.372836    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 16 22:33:15.372012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.371979    2581 patch_prober.go:28] interesting pod/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 16 22:33:15.372456 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.372038    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 16 22:33:15.514815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.514790    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:15.579805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.579775    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca\") pod \"83e480fb-3074-42ce-9d89-9dc3fdc4af92\" (UID: \"83e480fb-3074-42ce-9d89-9dc3fdc4af92\") "
Jul 16 22:33:15.580100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.580075    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "83e480fb-3074-42ce-9d89-9dc3fdc4af92" (UID: "83e480fb-3074-42ce-9d89-9dc3fdc4af92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:15.680932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:15.680854    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/83e480fb-3074-42ce-9d89-9dc3fdc4af92-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:16.371742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:16.371699    2581 patch_prober.go:28] interesting pod/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 16 22:33:16.371932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:16.371760    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 16 22:33:16.372572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:16.372546    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh"
Jul 16 22:33:16.372911 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:16.372571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh" event={"ID":"83e480fb-3074-42ce-9d89-9dc3fdc4af92","Type":"ContainerDied","Data":"c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56"}
Jul 16 22:33:16.372911 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:16.372619    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c57f0b52a6d25a49ebb5e358f5aad6ada7879e6dbcd0f492ad99a73a6a478e56"
Jul 16 22:33:17.371686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:17.371648    2581 patch_prober.go:28] interesting pod/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused" start-of-body=
Jul 16 22:33:17.371878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:17.371704    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.84:15021/healthz/ready\": dial tcp 10.133.0.84:15021: connect: connection refused"
Jul 16 22:33:18.376857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:18.376825    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:18.377285 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:18.377081    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:18.377900 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:18.377883    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:22.155045 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:22.155017    2581 scope.go:117] "RemoveContainer" containerID="a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab"
Jul 16 22:33:23.050424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.050343    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:23.050689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.050675    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="83e480fb-3074-42ce-9d89-9dc3fdc4af92" containerName="revoke-keys"
Jul 16 22:33:23.050689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.050688    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="83e480fb-3074-42ce-9d89-9dc3fdc4af92" containerName="revoke-keys"
Jul 16 22:33:23.050843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.050755    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="83e480fb-3074-42ce-9d89-9dc3fdc4af92" containerName="revoke-keys"
Jul 16 22:33:23.052713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.052693    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:23.062947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.062922    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:23.138323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.138297    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-frrx4\" (UID: \"3d43bd1e-7182-4317-bb40-c691e7c782f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:23.239429 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.239391    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-frrx4\" (UID: \"3d43bd1e-7182-4317-bb40-c691e7c782f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:23.240141 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.240113    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-frrx4\" (UID: \"3d43bd1e-7182-4317-bb40-c691e7c782f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:23.362812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.362778    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:23.405404 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.405368    2581 generic.go:356] "Generic (PLEG): container finished" podID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerID="bb897a62a942ac37fc761d13f2b861aca92dac511bf702ddb3c3cc3a9d9ad70b" exitCode=6
Jul 16 22:33:23.405568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.405438    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" event={"ID":"d8aa11c3-7920-46cf-af51-ae259b43314f","Type":"ContainerDied","Data":"bb897a62a942ac37fc761d13f2b861aca92dac511bf702ddb3c3cc3a9d9ad70b"}
Jul 16 22:33:23.405568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.405500    2581 scope.go:117] "RemoveContainer" containerID="a796c8979ad782baeec3483e1175be9fff5e016e9a25408a06c3d5ed8e892eab"
Jul 16 22:33:23.405802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.405785    2581 scope.go:117] "RemoveContainer" containerID="bb897a62a942ac37fc761d13f2b861aca92dac511bf702ddb3c3cc3a9d9ad70b"
Jul 16 22:33:23.406074 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:23.406050    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b_odh-ai-gateway-infra(d8aa11c3-7920-46cf-af51-ae259b43314f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f"
Jul 16 22:33:23.704838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:23.704816    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:23.706604 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:23.706570    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d43bd1e_7182_4317_bb40_c691e7c782f6.slice/crio-0cd6311a377cd95e8189a16d16d3db1d1e92a233c8d3c49e6831702baabf6fbd WatchSource:0}: Error finding container 0cd6311a377cd95e8189a16d16d3db1d1e92a233c8d3c49e6831702baabf6fbd: Status 404 returned error can't find the container with id 0cd6311a377cd95e8189a16d16d3db1d1e92a233c8d3c49e6831702baabf6fbd
Jul 16 22:33:24.413306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.413270    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerID="0c0bd00afe54a2df0ee79f9b2ee9eda83dba4c322bcfc94967f5327cca4b86d0" exitCode=6
Jul 16 22:33:24.413723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.413366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerDied","Data":"0c0bd00afe54a2df0ee79f9b2ee9eda83dba4c322bcfc94967f5327cca4b86d0"}
Jul 16 22:33:24.413723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.413437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerStarted","Data":"0cd6311a377cd95e8189a16d16d3db1d1e92a233c8d3c49e6831702baabf6fbd"}
Jul 16 22:33:24.413723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.413566    2581 scope.go:117] "RemoveContainer" containerID="0c0bd00afe54a2df0ee79f9b2ee9eda83dba4c322bcfc94967f5327cca4b86d0"
Jul 16 22:33:24.437391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.437368    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:24.573353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.573332    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:24.652292 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.652269    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca\") pod \"d8aa11c3-7920-46cf-af51-ae259b43314f\" (UID: \"d8aa11c3-7920-46cf-af51-ae259b43314f\") "
Jul 16 22:33:24.652635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.652615    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d8aa11c3-7920-46cf-af51-ae259b43314f" (UID: "d8aa11c3-7920-46cf-af51-ae259b43314f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:24.754068 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:24.754045    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d8aa11c3-7920-46cf-af51-ae259b43314f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:25.419484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.419400    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"
Jul 16 22:33:25.419484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.419406    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b" event={"ID":"d8aa11c3-7920-46cf-af51-ae259b43314f","Type":"ContainerDied","Data":"ef241c4413e466a33c5c4e644f7dc3b23ee633143cad0851961300faa98d3cfc"}
Jul 16 22:33:25.419484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.419477    2581 scope.go:117] "RemoveContainer" containerID="bb897a62a942ac37fc761d13f2b861aca92dac511bf702ddb3c3cc3a9d9ad70b"
Jul 16 22:33:25.421558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.421529    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerID="ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5" exitCode=6
Jul 16 22:33:25.421759 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.421623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerDied","Data":"ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5"}
Jul 16 22:33:25.422051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.422036    2581 scope.go:117] "RemoveContainer" containerID="ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5"
Jul 16 22:33:25.422251 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:25.422214    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-frrx4_odh-ai-gateway-infra(3d43bd1e-7182-4317-bb40-c691e7c782f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6"
Jul 16 22:33:25.429856 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.429838    2581 scope.go:117] "RemoveContainer" containerID="0c0bd00afe54a2df0ee79f9b2ee9eda83dba4c322bcfc94967f5327cca4b86d0"
Jul 16 22:33:25.439212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.439185    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:25.442544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:25.442523    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6g76b"]
Jul 16 22:33:26.202419 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:26.202384    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:33:26.428678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:26.428645    2581 scope.go:117] "RemoveContainer" containerID="ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5"
Jul 16 22:33:26.429069 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:26.428831    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-frrx4_odh-ai-gateway-infra(3d43bd1e-7182-4317-bb40-c691e7c782f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6"
Jul 16 22:33:27.161290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:27.161256    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" path="/var/lib/kubelet/pods/d8aa11c3-7920-46cf-af51-ae259b43314f/volumes"
Jul 16 22:33:37.155087 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:37.155053    2581 scope.go:117] "RemoveContainer" containerID="ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5"
Jul 16 22:33:37.479696 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:37.479667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerStarted","Data":"f6bfd051a5ad2a4e3c2d29fda808bd08fc2aef50b9599dc46e31beba23fbcf74"}
Jul 16 22:33:37.499552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:37.499501    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" podStartSLOduration=14.499485369 podStartE2EDuration="14.499485369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:37.498157695 +0000 UTC m=+2804.827597597" watchObservedRunningTime="2026-07-16 22:33:37.499485369 +0000 UTC m=+2804.828925272"
Jul 16 22:33:38.349750 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.349717    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:33:38.350178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.349932    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy" containerID="cri-o://0db9ee7b96c92d97dd3bc11dd445cb0540f2a774717422544e4aff127483dad8" gracePeriod=30
Jul 16 22:33:38.486168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.486132    2581 generic.go:356] "Generic (PLEG): container finished" podID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerID="f6bfd051a5ad2a4e3c2d29fda808bd08fc2aef50b9599dc46e31beba23fbcf74" exitCode=6
Jul 16 22:33:38.486376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.486208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerDied","Data":"f6bfd051a5ad2a4e3c2d29fda808bd08fc2aef50b9599dc46e31beba23fbcf74"}
Jul 16 22:33:38.486376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.486293    2581 scope.go:117] "RemoveContainer" containerID="ff9540cd5d9ceae4058af770f57fe972dc88d8cb1262b155d769dd9e3f7697c5"
Jul 16 22:33:38.486644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.486628    2581 scope.go:117] "RemoveContainer" containerID="f6bfd051a5ad2a4e3c2d29fda808bd08fc2aef50b9599dc46e31beba23fbcf74"
Jul 16 22:33:38.486873 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:38.486848    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-frrx4_odh-ai-gateway-infra(3d43bd1e-7182-4317-bb40-c691e7c782f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6"
Jul 16 22:33:38.513405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:38.513381    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:39.629274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:39.629250    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:39.678351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:39.678327    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca\") pod \"3d43bd1e-7182-4317-bb40-c691e7c782f6\" (UID: \"3d43bd1e-7182-4317-bb40-c691e7c782f6\") "
Jul 16 22:33:39.678679 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:39.678655    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3d43bd1e-7182-4317-bb40-c691e7c782f6" (UID: "3d43bd1e-7182-4317-bb40-c691e7c782f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:39.779788 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:39.779721    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3d43bd1e-7182-4317-bb40-c691e7c782f6-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:40.497960 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:40.497932    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"
Jul 16 22:33:40.498160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:40.497962    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4" event={"ID":"3d43bd1e-7182-4317-bb40-c691e7c782f6","Type":"ContainerDied","Data":"0cd6311a377cd95e8189a16d16d3db1d1e92a233c8d3c49e6831702baabf6fbd"}
Jul 16 22:33:40.498160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:40.498028    2581 scope.go:117] "RemoveContainer" containerID="f6bfd051a5ad2a4e3c2d29fda808bd08fc2aef50b9599dc46e31beba23fbcf74"
Jul 16 22:33:40.524112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:40.524084    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:40.526386 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:40.526362    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-frrx4"]
Jul 16 22:33:41.165122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:41.161982    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" path="/var/lib/kubelet/pods/3d43bd1e-7182-4317-bb40-c691e7c782f6/volumes"
Jul 16 22:33:43.514570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.514536    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerID="0db9ee7b96c92d97dd3bc11dd445cb0540f2a774717422544e4aff127483dad8" exitCode=0
Jul 16 22:33:43.514949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.514610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerDied","Data":"0db9ee7b96c92d97dd3bc11dd445cb0540f2a774717422544e4aff127483dad8"}
Jul 16 22:33:43.514949 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.514676    2581 scope.go:117] "RemoveContainer" containerID="e8e8a65403861eaf532f2ed908a1a295a171b2816e4cd008ee37108814a195f2"
Jul 16 22:33:43.608174 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.608149    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:43.713331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713312    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713329    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713361    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lhxpp\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713408    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713440    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713467    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713581    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:43.713860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713582    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713686    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token\") pod \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\" (UID: \"5a5831ec-2a59-4294-ba9f-8031da33d8cd\") "
Jul 16 22:33:43.713860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713689    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:43.713860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:43.714078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.713913    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:43.714078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.714018    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.714078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.714038    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.714078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.714053    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.714078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.714067    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.714549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.714101    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data" (OuterVolumeSpecName: "istio-data") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:43.715891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.715859    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:33:43.716009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.715896    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token" (OuterVolumeSpecName: "istio-token") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:33:43.716009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.715933    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:33:43.716009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.715962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp" (OuterVolumeSpecName: "kube-api-access-lhxpp") pod "5a5831ec-2a59-4294-ba9f-8031da33d8cd" (UID: "5a5831ec-2a59-4294-ba9f-8031da33d8cd"). InnerVolumeSpecName "kube-api-access-lhxpp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:33:43.814815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.814778    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.814815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.814808    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.814815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.814819    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhxpp\" (UniqueName: \"kubernetes.io/projected/5a5831ec-2a59-4294-ba9f-8031da33d8cd-kube-api-access-lhxpp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.815059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.814828    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:43.815059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:43.814835    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a5831ec-2a59-4294-ba9f-8031da33d8cd-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:33:44.520573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.520530    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb" event={"ID":"5a5831ec-2a59-4294-ba9f-8031da33d8cd","Type":"ContainerDied","Data":"ccbf52bdd225099f521419eb8f4073a54eb1a0553ed9e4e875e5263a70e2723a"}
Jul 16 22:33:44.521000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.520599    2581 scope.go:117] "RemoveContainer" containerID="0db9ee7b96c92d97dd3bc11dd445cb0540f2a774717422544e4aff127483dad8"
Jul 16 22:33:44.521000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.520550    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"
Jul 16 22:33:44.551509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.551478    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:33:44.559928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.559902    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-50ab3915-openshift-default-58b7d59d79-wznvb"]
Jul 16 22:33:44.634573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634547    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:33:44.634850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634838    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.634893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634851    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.634893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634864    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:44.634893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634870    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:44.634893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634879    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.634893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634894    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634900    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634906    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634912    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634919    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634924    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634934    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634940    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634990    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.634997    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.635002    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.635010    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:44.635047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.635015    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:44.639094 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.639079    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:33:44.643182 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.643158    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:33:44.643182 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.643174    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:44.664054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.664029    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:33:44.722436 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.722412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2\" (UID: \"3a96cf4f-8707-48f0-868b-0d93686d3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:33:44.823774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.823698    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2\" (UID: \"3a96cf4f-8707-48f0-868b-0d93686d3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:33:44.824293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.824274    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2\" (UID: \"3a96cf4f-8707-48f0-868b-0d93686d3105\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:33:44.948599 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:44.948580    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:33:45.088219 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:45.088190    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:33:45.092130 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:45.092094    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a96cf4f_8707_48f0_868b_0d93686d3105.slice/crio-7028c8f37459715ae54aa957dae42b48746b7cdb33516c471b27ee047140a2a9 WatchSource:0}: Error finding container 7028c8f37459715ae54aa957dae42b48746b7cdb33516c471b27ee047140a2a9: Status 404 returned error can't find the container with id 7028c8f37459715ae54aa957dae42b48746b7cdb33516c471b27ee047140a2a9
Jul 16 22:33:45.160508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:45.160479    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" path="/var/lib/kubelet/pods/5a5831ec-2a59-4294-ba9f-8031da33d8cd/volumes"
Jul 16 22:33:45.528267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:45.528208    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerStarted","Data":"e926006aa932ba890718ca3520e38a897b890274b368527bb202dcac4316685d"}
Jul 16 22:33:45.528716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:45.528297    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerStarted","Data":"7028c8f37459715ae54aa957dae42b48746b7cdb33516c471b27ee047140a2a9"}
Jul 16 22:33:45.545099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:45.545044    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" podStartSLOduration=1.545027128 podStartE2EDuration="1.545027128s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:45.544315037 +0000 UTC m=+2812.873754940" watchObservedRunningTime="2026-07-16 22:33:45.545027128 +0000 UTC m=+2812.874467032"
Jul 16 22:33:46.535146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:46.535114    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerID="e926006aa932ba890718ca3520e38a897b890274b368527bb202dcac4316685d" exitCode=6
Jul 16 22:33:46.535564 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:46.535169    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerDied","Data":"e926006aa932ba890718ca3520e38a897b890274b368527bb202dcac4316685d"}
Jul 16 22:33:46.535564 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:46.535368    2581 scope.go:117] "RemoveContainer" containerID="e926006aa932ba890718ca3520e38a897b890274b368527bb202dcac4316685d"
Jul 16 22:33:47.541482 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:47.541396    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerID="4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad" exitCode=6
Jul 16 22:33:47.541883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:47.541481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerDied","Data":"4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad"}
Jul 16 22:33:47.541883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:47.541537    2581 scope.go:117] "RemoveContainer" containerID="e926006aa932ba890718ca3520e38a897b890274b368527bb202dcac4316685d"
Jul 16 22:33:47.541883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:47.541624    2581 scope.go:117] "RemoveContainer" containerID="4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad"
Jul 16 22:33:47.541883 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:47.541852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2_odh-ai-gateway-infra(3a96cf4f-8707-48f0-868b-0d93686d3105)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105"
Jul 16 22:33:48.548078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:48.548048    2581 scope.go:117] "RemoveContainer" containerID="4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad"
Jul 16 22:33:48.548532 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:48.548300    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2_odh-ai-gateway-infra(3a96cf4f-8707-48f0-868b-0d93686d3105)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105"
Jul 16 22:33:57.037718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.037683    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:33:57.038186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.038147    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:57.038186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.038166    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:57.038345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.038274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3d43bd1e-7182-4317-bb40-c691e7c782f6" containerName="revoke-keys"
Jul 16 22:33:57.038345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.038288    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d8aa11c3-7920-46cf-af51-ae259b43314f" containerName="revoke-keys"
Jul 16 22:33:57.038345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.038298    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a5831ec-2a59-4294-ba9f-8031da33d8cd" containerName="istio-proxy"
Jul 16 22:33:57.042517 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.042495    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:33:57.056608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.056581    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:33:57.131628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.131600    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7\" (UID: \"73a5307d-dcc2-4701-8408-6c6c082ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:33:57.232572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.232540    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7\" (UID: \"73a5307d-dcc2-4701-8408-6c6c082ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:33:57.233076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.233056    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7\" (UID: \"73a5307d-dcc2-4701-8408-6c6c082ac499\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:33:57.351772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.351740    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:33:57.490294 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.490263    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:33:57.493631 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:57.493602    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a5307d_dcc2_4701_8408_6c6c082ac499.slice/crio-7a6293a80185a7fa4675bfdb7ecca6adff286b7ff43654b552bf8ff478cc1052 WatchSource:0}: Error finding container 7a6293a80185a7fa4675bfdb7ecca6adff286b7ff43654b552bf8ff478cc1052: Status 404 returned error can't find the container with id 7a6293a80185a7fa4675bfdb7ecca6adff286b7ff43654b552bf8ff478cc1052
Jul 16 22:33:57.589905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:57.589873    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerStarted","Data":"7a6293a80185a7fa4675bfdb7ecca6adff286b7ff43654b552bf8ff478cc1052"}
Jul 16 22:33:58.594924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:58.594888    2581 generic.go:356] "Generic (PLEG): container finished" podID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerID="4c1e359b1c5af622311933a08f3f31075894743de2fa85929173ff79ea282be7" exitCode=6
Jul 16 22:33:58.595450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:58.594940    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerDied","Data":"4c1e359b1c5af622311933a08f3f31075894743de2fa85929173ff79ea282be7"}
Jul 16 22:33:58.595450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:58.595182    2581 scope.go:117] "RemoveContainer" containerID="4c1e359b1c5af622311933a08f3f31075894743de2fa85929173ff79ea282be7"
Jul 16 22:33:59.463332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.463299    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:33:59.466567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.466551    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.469734 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.469715    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-b3fb3ce5-openshift-default-dockercfg-w427j\""
Jul 16 22:33:59.510464 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.510391    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:33:59.554048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554007    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554253 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554098    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554253 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554136    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554253 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554182    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554253 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554210    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554262    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554305    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554336    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.554478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.554406    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.600845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.600809    2581 generic.go:356] "Generic (PLEG): container finished" podID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerID="1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32" exitCode=6
Jul 16 22:33:59.601315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.600852    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerDied","Data":"1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32"}
Jul 16 22:33:59.601315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.600902    2581 scope.go:117] "RemoveContainer" containerID="4c1e359b1c5af622311933a08f3f31075894743de2fa85929173ff79ea282be7"
Jul 16 22:33:59.601315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.601103    2581 scope.go:117] "RemoveContainer" containerID="1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32"
Jul 16 22:33:59.601474 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:33:59.601361    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7_odh-ai-gateway-infra(73a5307d-dcc2-4701-8408-6c6c082ac499)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499"
Jul 16 22:33:59.655163 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655118    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655184    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655271    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655313    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655405    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655451    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655484    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655511    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655636    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655769    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.655887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.655856    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.656343 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.656316    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.657975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.657955    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.658155 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.658137    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.667401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.667360    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.667672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.667647    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x\") pod \"e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") " pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.776374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.776283    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:33:59.936275 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.936250    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:33:59.938363 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:33:59.938338    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4de4181_ff14_4c1f_b673_b13debcf2fef.slice/crio-c95d87a506de83c326d664d42e18cf013262d76e6c9da91726d76f1864046917 WatchSource:0}: Error finding container c95d87a506de83c326d664d42e18cf013262d76e6c9da91726d76f1864046917: Status 404 returned error can't find the container with id c95d87a506de83c326d664d42e18cf013262d76e6c9da91726d76f1864046917
Jul 16 22:33:59.940822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.940789    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:33:59.940907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.940856    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:33:59.940907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:33:59.940885    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:34:00.606932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.606893    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerStarted","Data":"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"}
Jul 16 22:34:00.607411 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.606950    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerStarted","Data":"c95d87a506de83c326d664d42e18cf013262d76e6c9da91726d76f1864046917"}
Jul 16 22:34:00.608581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.608564    2581 scope.go:117] "RemoveContainer" containerID="1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32"
Jul 16 22:34:00.608757 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:00.608739    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7_odh-ai-gateway-infra(73a5307d-dcc2-4701-8408-6c6c082ac499)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499"
Jul 16 22:34:00.634748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.634700    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podStartSLOduration=1.634685425 podStartE2EDuration="1.634685425s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:00.633650807 +0000 UTC m=+2827.963090709" watchObservedRunningTime="2026-07-16 22:34:00.634685425 +0000 UTC m=+2827.964125327"
Jul 16 22:34:00.776454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.776427    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:00.781047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:00.781024    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:01.613870 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:01.613838    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:01.614802 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:01.614779    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:02.155493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:02.155463    2581 scope.go:117] "RemoveContainer" containerID="4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad"
Jul 16 22:34:02.620869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:02.620829    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerStarted","Data":"ae51cdf59eeeb9ddb5ffd4bb3d9b5176f105bc0a29c4798ab6087184993b52de"}
Jul 16 22:34:03.630641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:03.630608    2581 generic.go:356] "Generic (PLEG): container finished" podID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerID="ae51cdf59eeeb9ddb5ffd4bb3d9b5176f105bc0a29c4798ab6087184993b52de" exitCode=6
Jul 16 22:34:03.631070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:03.630677    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerDied","Data":"ae51cdf59eeeb9ddb5ffd4bb3d9b5176f105bc0a29c4798ab6087184993b52de"}
Jul 16 22:34:03.631070 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:03.630737    2581 scope.go:117] "RemoveContainer" containerID="4d6b9c50375ecc698c80a8aa3c34dcdb578ca1fea789da872df48269abc65aad"
Jul 16 22:34:03.631198 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:03.631106    2581 scope.go:117] "RemoveContainer" containerID="ae51cdf59eeeb9ddb5ffd4bb3d9b5176f105bc0a29c4798ab6087184993b52de"
Jul 16 22:34:03.631344 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:03.631319    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2_odh-ai-gateway-infra(3a96cf4f-8707-48f0-868b-0d93686d3105)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105"
Jul 16 22:34:03.694524 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:03.694489    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:34:04.758465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:04.758440    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:34:04.903083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:04.902993    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca\") pod \"3a96cf4f-8707-48f0-868b-0d93686d3105\" (UID: \"3a96cf4f-8707-48f0-868b-0d93686d3105\") "
Jul 16 22:34:04.903428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:04.903406    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a96cf4f-8707-48f0-868b-0d93686d3105" (UID: "3a96cf4f-8707-48f0-868b-0d93686d3105"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:05.003970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.003933    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a96cf4f-8707-48f0-868b-0d93686d3105-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:34:05.641522 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.641490    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"
Jul 16 22:34:05.641731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.641517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2" event={"ID":"3a96cf4f-8707-48f0-868b-0d93686d3105","Type":"ContainerDied","Data":"7028c8f37459715ae54aa957dae42b48746b7cdb33516c471b27ee047140a2a9"}
Jul 16 22:34:05.641731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.641582    2581 scope.go:117] "RemoveContainer" containerID="ae51cdf59eeeb9ddb5ffd4bb3d9b5176f105bc0a29c4798ab6087184993b52de"
Jul 16 22:34:05.670587 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.670564    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:34:05.673521 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:05.673499    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvtn2"]
Jul 16 22:34:07.159698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:07.159666    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" path="/var/lib/kubelet/pods/3a96cf4f-8707-48f0-868b-0d93686d3105/volumes"
Jul 16 22:34:09.205145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:09.205108    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:34:12.154871 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:12.154837    2581 scope.go:117] "RemoveContainer" containerID="1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32"
Jul 16 22:34:12.678416 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:12.678356    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerStarted","Data":"66ca7552f19c3ee72a22a3be526ce3a3e2596b96df052065d598425ac7949101"}
Jul 16 22:34:12.703138 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:12.703093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" podStartSLOduration=15.70307815 podStartE2EDuration="15.70307815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:12.70043312 +0000 UTC m=+2840.029873025" watchObservedRunningTime="2026-07-16 22:34:12.70307815 +0000 UTC m=+2840.032518052"
Jul 16 22:34:13.095989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.095952    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:13.096304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096292    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096306    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096318    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096323    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096339    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096344    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096397    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096403    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.096530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.096409    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a96cf4f-8707-48f0-868b-0d93686d3105" containerName="revoke-keys"
Jul 16 22:34:13.099261 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.099245    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:13.112589 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.112569    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:13.176330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.176301    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-dksg5\" (UID: \"2f202082-6e3f-478c-ac49-3e211fa282fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:13.277728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.277643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-dksg5\" (UID: \"2f202082-6e3f-478c-ac49-3e211fa282fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:13.278218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.278198    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-dksg5\" (UID: \"2f202082-6e3f-478c-ac49-3e211fa282fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:13.408139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.408107    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:13.684100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.684069    2581 generic.go:356] "Generic (PLEG): container finished" podID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerID="66ca7552f19c3ee72a22a3be526ce3a3e2596b96df052065d598425ac7949101" exitCode=6
Jul 16 22:34:13.684300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.684143    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerDied","Data":"66ca7552f19c3ee72a22a3be526ce3a3e2596b96df052065d598425ac7949101"}
Jul 16 22:34:13.684300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.684201    2581 scope.go:117] "RemoveContainer" containerID="1da449611acbdadfd3138768288d84151d85e28a0533f7f5ad9dd495e977cd32"
Jul 16 22:34:13.684548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.684531    2581 scope.go:117] "RemoveContainer" containerID="66ca7552f19c3ee72a22a3be526ce3a3e2596b96df052065d598425ac7949101"
Jul 16 22:34:13.684771 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:13.684751    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7_odh-ai-gateway-infra(73a5307d-dcc2-4701-8408-6c6c082ac499)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499"
Jul 16 22:34:13.722538 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.722505    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:34:13.818112 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:13.818091    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:13.820013 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:34:13.819987    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f202082_6e3f_478c_ac49_3e211fa282fa.slice/crio-87f6480f1367e38daa1daaba99c508e0d5942cd8cd25bba099074bb2211ae9de WatchSource:0}: Error finding container 87f6480f1367e38daa1daaba99c508e0d5942cd8cd25bba099074bb2211ae9de: Status 404 returned error can't find the container with id 87f6480f1367e38daa1daaba99c508e0d5942cd8cd25bba099074bb2211ae9de
Jul 16 22:34:14.692510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.692460    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerID="30805b825e3179b2112a737ff78745f6e0287efbbd97fc2b71667116c114c5be" exitCode=6
Jul 16 22:34:14.692910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.692529    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" event={"ID":"2f202082-6e3f-478c-ac49-3e211fa282fa","Type":"ContainerDied","Data":"30805b825e3179b2112a737ff78745f6e0287efbbd97fc2b71667116c114c5be"}
Jul 16 22:34:14.692910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.692590    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" event={"ID":"2f202082-6e3f-478c-ac49-3e211fa282fa","Type":"ContainerStarted","Data":"87f6480f1367e38daa1daaba99c508e0d5942cd8cd25bba099074bb2211ae9de"}
Jul 16 22:34:14.692910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.692690    2581 scope.go:117] "RemoveContainer" containerID="30805b825e3179b2112a737ff78745f6e0287efbbd97fc2b71667116c114c5be"
Jul 16 22:34:14.841086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.841056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:34:14.890140 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.890108    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca\") pod \"73a5307d-dcc2-4701-8408-6c6c082ac499\" (UID: \"73a5307d-dcc2-4701-8408-6c6c082ac499\") "
Jul 16 22:34:14.890484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.890464    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "73a5307d-dcc2-4701-8408-6c6c082ac499" (UID: "73a5307d-dcc2-4701-8408-6c6c082ac499"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:14.991604 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:14.991571    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73a5307d-dcc2-4701-8408-6c6c082ac499-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:34:15.698040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.698006    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"
Jul 16 22:34:15.698530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.698037    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7" event={"ID":"73a5307d-dcc2-4701-8408-6c6c082ac499","Type":"ContainerDied","Data":"7a6293a80185a7fa4675bfdb7ecca6adff286b7ff43654b552bf8ff478cc1052"}
Jul 16 22:34:15.698530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.698092    2581 scope.go:117] "RemoveContainer" containerID="66ca7552f19c3ee72a22a3be526ce3a3e2596b96df052065d598425ac7949101"
Jul 16 22:34:15.700668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.700640    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerID="69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56" exitCode=6
Jul 16 22:34:15.700793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.700675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" event={"ID":"2f202082-6e3f-478c-ac49-3e211fa282fa","Type":"ContainerDied","Data":"69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56"}
Jul 16 22:34:15.700974 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.700960    2581 scope.go:117] "RemoveContainer" containerID="69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56"
Jul 16 22:34:15.701171 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:15.701152    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-dksg5_odh-ai-gateway-infra(2f202082-6e3f-478c-ac49-3e211fa282fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa"
Jul 16 22:34:15.708144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.708127    2581 scope.go:117] "RemoveContainer" containerID="30805b825e3179b2112a737ff78745f6e0287efbbd97fc2b71667116c114c5be"
Jul 16 22:34:15.714643 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.714623    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:34:15.720305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:15.720272    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-k4nj7"]
Jul 16 22:34:16.098146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098112    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:34:16.098508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098492    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098511    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098528    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098535    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098550    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098557    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098656    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.098845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.098670    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:16.103536 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.103516    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.106393 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.106373    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b3fb3ce5\"/\"kube-root-ca.crt\""
Jul 16 22:34:16.106489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.106373    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b3fb3ce5\"/\"e2e-lifecycle-model-b3fb3ce5-kserve-self-signed-certs\""
Jul 16 22:34:16.106821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.106803    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b3fb3ce5\"/\"openshift-service-ca.crt\""
Jul 16 22:34:16.106919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.106805    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b3fb3ce5\"/\"default-dockercfg-r8m5j\""
Jul 16 22:34:16.113212 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.113187    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:34:16.201594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201557    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.201794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201606    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dgrc\" (UniqueName: \"kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.201794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201661    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.201794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201740    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.201937 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201814    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.201937 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.201919    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303060 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303028    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303068    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dgrc\" (UniqueName: \"kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303120    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303153    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303196    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303520 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303265    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303584    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303643    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.303722 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.303682    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.305620 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.305599    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.305892 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.305872    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.312075 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.312055    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dgrc\" (UniqueName: \"kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc\") pod \"e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") " pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.413414 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.413333    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:16.551835 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.550828    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:34:16.553618 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:34:16.553574    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa9617d3_f37f_4d7f_9c96_799677bb13da.slice/crio-deb427a11436cb406a232ab69cc41caa37287f9f87746ee35b753f1f4c67362e WatchSource:0}: Error finding container deb427a11436cb406a232ab69cc41caa37287f9f87746ee35b753f1f4c67362e: Status 404 returned error can't find the container with id deb427a11436cb406a232ab69cc41caa37287f9f87746ee35b753f1f4c67362e
Jul 16 22:34:16.555742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.555726    2581 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:34:16.614775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.614744    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 16 22:34:16.614906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.614800    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 16 22:34:16.709329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.709210    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" event={"ID":"aa9617d3-f37f-4d7f-9c96-799677bb13da","Type":"ContainerStarted","Data":"deb427a11436cb406a232ab69cc41caa37287f9f87746ee35b753f1f4c67362e"}
Jul 16 22:34:16.711152 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.711125    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerID="83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2" exitCode=0
Jul 16 22:34:16.711319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.711190    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerDied","Data":"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"}
Jul 16 22:34:16.711711 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.711690    2581 scope.go:117] "RemoveContainer" containerID="83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"
Jul 16 22:34:16.712811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.712778    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:34:16.712905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.712868    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:34:16.712968 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.712919    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:34:16.714172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:16.714154    2581 scope.go:117] "RemoveContainer" containerID="69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56"
Jul 16 22:34:16.714385 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:16.714359    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-dksg5_odh-ai-gateway-infra(2f202082-6e3f-478c-ac49-3e211fa282fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa"
Jul 16 22:34:17.162498 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.162467    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" path="/var/lib/kubelet/pods/73a5307d-dcc2-4701-8408-6c6c082ac499/volumes"
Jul 16 22:34:17.720106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.720066    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" event={"ID":"aa9617d3-f37f-4d7f-9c96-799677bb13da","Type":"ContainerStarted","Data":"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"}
Jul 16 22:34:17.722255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.722212    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerStarted","Data":"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"}
Jul 16 22:34:17.741667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.741626    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" podStartSLOduration=0.670307402 podStartE2EDuration="1.741611084s" totalImagesPullingTime="1.071303682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:17.740612939 +0000 UTC m=+2845.070052840" watchObservedRunningTime="2026-07-16 22:34:17.741611084 +0000 UTC m=+2845.071050986"
Jul 16 22:34:17.776656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.776622    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:17.776890 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.776865    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 16 22:34:17.776961 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:17.776926    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 16 22:34:18.776700 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:18.776668    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 16 22:34:18.777088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:18.776724    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 16 22:34:19.777152 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:19.777117    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 16 22:34:19.777567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:19.777178    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 16 22:34:20.707204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:20.707155    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:34:20.776874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:20.776840    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused" start-of-body=
Jul 16 22:34:20.777036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:20.776893    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.90:15021/healthz/ready\": dial tcp 10.133.0.90:15021: connect: connection refused"
Jul 16 22:34:21.781549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:21.781511    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:21.782035 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:21.781926    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:21.782695 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:21.782673    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:34:25.488654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.488616    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:25.489059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.489016    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="73a5307d-dcc2-4701-8408-6c6c082ac499" containerName="revoke-keys"
Jul 16 22:34:25.491606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.491585    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:25.502178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.502145    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:25.607532 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.607494    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d\" (UID: \"c56090f5-34ca-41e5-8a66-4d4ee2aa1188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:25.709049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.709000    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d\" (UID: \"c56090f5-34ca-41e5-8a66-4d4ee2aa1188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:25.709669 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.709648    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d\" (UID: \"c56090f5-34ca-41e5-8a66-4d4ee2aa1188\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:25.803603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.803492    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:25.946936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:25.946900    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:25.951062 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:34:25.951033    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc56090f5_34ca_41e5_8a66_4d4ee2aa1188.slice/crio-5aaed57d868309ac5e57da19a6a0796c0709ac2fc14e36f7c85b50510d0bd1f5 WatchSource:0}: Error finding container 5aaed57d868309ac5e57da19a6a0796c0709ac2fc14e36f7c85b50510d0bd1f5: Status 404 returned error can't find the container with id 5aaed57d868309ac5e57da19a6a0796c0709ac2fc14e36f7c85b50510d0bd1f5
Jul 16 22:34:26.414186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.414082    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:26.414186 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.414152    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:26.427306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.427281    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:26.767570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.767470    2581 generic.go:356] "Generic (PLEG): container finished" podID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerID="e06ff1275d19a1b69b9ee39f620a06243b7a342fc16dbd164e8dbd661bdf95c0" exitCode=6
Jul 16 22:34:26.767570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.767543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerDied","Data":"e06ff1275d19a1b69b9ee39f620a06243b7a342fc16dbd164e8dbd661bdf95c0"}
Jul 16 22:34:26.768016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.767638    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerStarted","Data":"5aaed57d868309ac5e57da19a6a0796c0709ac2fc14e36f7c85b50510d0bd1f5"}
Jul 16 22:34:26.768016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.767757    2581 scope.go:117] "RemoveContainer" containerID="e06ff1275d19a1b69b9ee39f620a06243b7a342fc16dbd164e8dbd661bdf95c0"
Jul 16 22:34:26.780158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:26.780137    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:34:27.773756 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:27.773665    2581 generic.go:356] "Generic (PLEG): container finished" podID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerID="7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89" exitCode=6
Jul 16 22:34:27.773756 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:27.773739    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerDied","Data":"7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89"}
Jul 16 22:34:27.774260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:27.773786    2581 scope.go:117] "RemoveContainer" containerID="e06ff1275d19a1b69b9ee39f620a06243b7a342fc16dbd164e8dbd661bdf95c0"
Jul 16 22:34:27.774260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:27.773973    2581 scope.go:117] "RemoveContainer" containerID="7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89"
Jul 16 22:34:27.774260 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:27.774197    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d_odh-ai-gateway-infra(c56090f5-34ca-41e5-8a66-4d4ee2aa1188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188"
Jul 16 22:34:28.780393 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:28.780366    2581 scope.go:117] "RemoveContainer" containerID="7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89"
Jul 16 22:34:28.780780 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:28.780549    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d_odh-ai-gateway-infra(c56090f5-34ca-41e5-8a66-4d4ee2aa1188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188"
Jul 16 22:34:29.154555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:29.154526    2581 scope.go:117] "RemoveContainer" containerID="69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56"
Jul 16 22:34:29.786123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:29.786092    2581 generic.go:356] "Generic (PLEG): container finished" podID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerID="c31f1255ed391ab80bb0c2b0315e299578390b84af3eaea853577f6b5d278df4" exitCode=6
Jul 16 22:34:29.786471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:29.786170    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" event={"ID":"2f202082-6e3f-478c-ac49-3e211fa282fa","Type":"ContainerDied","Data":"c31f1255ed391ab80bb0c2b0315e299578390b84af3eaea853577f6b5d278df4"}
Jul 16 22:34:29.786471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:29.786248    2581 scope.go:117] "RemoveContainer" containerID="69a6057d801fe4766cd8919f233f3c56bcb9fe693b7c7eed8bf0f9132025cb56"
Jul 16 22:34:29.786582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:29.786538    2581 scope.go:117] "RemoveContainer" containerID="c31f1255ed391ab80bb0c2b0315e299578390b84af3eaea853577f6b5d278df4"
Jul 16 22:34:29.786768 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:29.786747    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-dksg5_odh-ai-gateway-infra(2f202082-6e3f-478c-ac49-3e211fa282fa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa"
Jul 16 22:34:30.825570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:30.825541    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:30.966817 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:30.966793    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:31.055712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.055682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca\") pod \"2f202082-6e3f-478c-ac49-3e211fa282fa\" (UID: \"2f202082-6e3f-478c-ac49-3e211fa282fa\") "
Jul 16 22:34:31.056002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.055982    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2f202082-6e3f-478c-ac49-3e211fa282fa" (UID: "2f202082-6e3f-478c-ac49-3e211fa282fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:31.157030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.157003    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f202082-6e3f-478c-ac49-3e211fa282fa-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:34:31.799106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.799075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"
Jul 16 22:34:31.799310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.799122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5" event={"ID":"2f202082-6e3f-478c-ac49-3e211fa282fa","Type":"ContainerDied","Data":"87f6480f1367e38daa1daaba99c508e0d5942cd8cd25bba099074bb2211ae9de"}
Jul 16 22:34:31.799310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.799202    2581 scope.go:117] "RemoveContainer" containerID="c31f1255ed391ab80bb0c2b0315e299578390b84af3eaea853577f6b5d278df4"
Jul 16 22:34:31.818194 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.818107    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:31.820127 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:31.820107    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-dksg5"]
Jul 16 22:34:33.159785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:33.159750    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" path="/var/lib/kubelet/pods/2f202082-6e3f-478c-ac49-3e211fa282fa/volumes"
Jul 16 22:34:42.154727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:42.154693    2581 scope.go:117] "RemoveContainer" containerID="7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89"
Jul 16 22:34:42.858719 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:42.858673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerStarted","Data":"1cead7891a3dbed7687b557dfb536825f0fea01ffa83e275cd2920f872012c43"}
Jul 16 22:34:42.887033 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:42.886986    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" podStartSLOduration=17.886971924 podStartE2EDuration="17.886971924s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:42.885653517 +0000 UTC m=+2870.215093423" watchObservedRunningTime="2026-07-16 22:34:42.886971924 +0000 UTC m=+2870.216411827"
Jul 16 22:34:43.864582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:43.864540    2581 generic.go:356] "Generic (PLEG): container finished" podID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerID="1cead7891a3dbed7687b557dfb536825f0fea01ffa83e275cd2920f872012c43" exitCode=6
Jul 16 22:34:43.865069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:43.864613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerDied","Data":"1cead7891a3dbed7687b557dfb536825f0fea01ffa83e275cd2920f872012c43"}
Jul 16 22:34:43.865069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:43.864676    2581 scope.go:117] "RemoveContainer" containerID="7f39c5b3ba82757fae31f708642f96f9854206908b998b7c7f939d4a57a41f89"
Jul 16 22:34:43.865069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:43.864966    2581 scope.go:117] "RemoveContainer" containerID="1cead7891a3dbed7687b557dfb536825f0fea01ffa83e275cd2920f872012c43"
Jul 16 22:34:43.865294 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:43.865169    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d_odh-ai-gateway-infra(c56090f5-34ca-41e5-8a66-4d4ee2aa1188)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188"
Jul 16 22:34:43.898096 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:43.898067    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:44.678902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.678870    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:34:44.679206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679194    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.679279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679208    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.679279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679217    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.679279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679222    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.679383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679299    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.679383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.679307    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:44.683645 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.683624    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:34:44.691084 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.691056    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:34:44.874491 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.874457    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz\" (UID: \"63157b2b-a51f-44ff-a005-5288b504d94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:34:44.975748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.975646    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz\" (UID: \"63157b2b-a51f-44ff-a005-5288b504d94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:34:44.976356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.976334    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz\" (UID: \"63157b2b-a51f-44ff-a005-5288b504d94f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:34:44.994996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:44.994971    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:34:45.024216 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.024191    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:45.137329 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.137207    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:34:45.139723 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:34:45.139697    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63157b2b_a51f_44ff_a005_5288b504d94f.slice/crio-1cf823399158ae805e93be7d6b2fd6f0546cc16150ec0de738089f3b122af430 WatchSource:0}: Error finding container 1cf823399158ae805e93be7d6b2fd6f0546cc16150ec0de738089f3b122af430: Status 404 returned error can't find the container with id 1cf823399158ae805e93be7d6b2fd6f0546cc16150ec0de738089f3b122af430
Jul 16 22:34:45.177591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.177559    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca\") pod \"c56090f5-34ca-41e5-8a66-4d4ee2aa1188\" (UID: \"c56090f5-34ca-41e5-8a66-4d4ee2aa1188\") "
Jul 16 22:34:45.177969 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.177945    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c56090f5-34ca-41e5-8a66-4d4ee2aa1188" (UID: "c56090f5-34ca-41e5-8a66-4d4ee2aa1188"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:45.279249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.279147    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c56090f5-34ca-41e5-8a66-4d4ee2aa1188-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:34:45.876082 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.876057    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"
Jul 16 22:34:45.876499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.876077    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d" event={"ID":"c56090f5-34ca-41e5-8a66-4d4ee2aa1188","Type":"ContainerDied","Data":"5aaed57d868309ac5e57da19a6a0796c0709ac2fc14e36f7c85b50510d0bd1f5"}
Jul 16 22:34:45.876499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.876138    2581 scope.go:117] "RemoveContainer" containerID="1cead7891a3dbed7687b557dfb536825f0fea01ffa83e275cd2920f872012c43"
Jul 16 22:34:45.877902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.877868    2581 generic.go:356] "Generic (PLEG): container finished" podID="63157b2b-a51f-44ff-a005-5288b504d94f" containerID="7e4abb5f1cf02561a9bb7c2fcb00b4a62d942895d9479c6dfb7af0cd4e7c4ee1" exitCode=6
Jul 16 22:34:45.878009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.877926    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" event={"ID":"63157b2b-a51f-44ff-a005-5288b504d94f","Type":"ContainerDied","Data":"7e4abb5f1cf02561a9bb7c2fcb00b4a62d942895d9479c6dfb7af0cd4e7c4ee1"}
Jul 16 22:34:45.878009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.877967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" event={"ID":"63157b2b-a51f-44ff-a005-5288b504d94f","Type":"ContainerStarted","Data":"1cf823399158ae805e93be7d6b2fd6f0546cc16150ec0de738089f3b122af430"}
Jul 16 22:34:45.878158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.878144    2581 scope.go:117] "RemoveContainer" containerID="7e4abb5f1cf02561a9bb7c2fcb00b4a62d942895d9479c6dfb7af0cd4e7c4ee1"
Jul 16 22:34:45.910535 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.910512    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:45.916169 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:45.916149    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-gtz9d"]
Jul 16 22:34:46.891163 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:46.891125    2581 generic.go:356] "Generic (PLEG): container finished" podID="63157b2b-a51f-44ff-a005-5288b504d94f" containerID="038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f" exitCode=6
Jul 16 22:34:46.891603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:46.891205    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" event={"ID":"63157b2b-a51f-44ff-a005-5288b504d94f","Type":"ContainerDied","Data":"038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f"}
Jul 16 22:34:46.891603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:46.891307    2581 scope.go:117] "RemoveContainer" containerID="7e4abb5f1cf02561a9bb7c2fcb00b4a62d942895d9479c6dfb7af0cd4e7c4ee1"
Jul 16 22:34:46.891603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:46.891494    2581 scope.go:117] "RemoveContainer" containerID="038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f"
Jul 16 22:34:46.891742 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:46.891686    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz_odh-ai-gateway-infra(63157b2b-a51f-44ff-a005-5288b504d94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" podUID="63157b2b-a51f-44ff-a005-5288b504d94f"
Jul 16 22:34:47.109389 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109356    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:34:47.109799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109779    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.109866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109803    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.109866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109830    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.109866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109838    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.109866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:47.109866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109857    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:47.110036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109921    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f202082-6e3f-478c-ac49-3e211fa282fa" containerName="revoke-keys"
Jul 16 22:34:47.110036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109929    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.110036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109937    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.110036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.109942    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:34:47.114244 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.114206    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:34:47.162277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.162197    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" path="/var/lib/kubelet/pods/c56090f5-34ca-41e5-8a66-4d4ee2aa1188/volumes"
Jul 16 22:34:47.162528 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.162514    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:34:47.295833 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.295798    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k\" (UID: \"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:34:47.397137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.397105    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k\" (UID: \"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:34:47.397721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.397702    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k\" (UID: \"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:34:47.423859 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.423788    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:34:47.563927 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.563903    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:34:47.565872 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:34:47.565845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd03e8e2f_24c7_41a1_b8f4_15ba73bf7bc2.slice/crio-dadf4c5567900a14ce6f319948fca823a74c435a48bba24bad224c05d928985e WatchSource:0}: Error finding container dadf4c5567900a14ce6f319948fca823a74c435a48bba24bad224c05d928985e: Status 404 returned error can't find the container with id dadf4c5567900a14ce6f319948fca823a74c435a48bba24bad224c05d928985e
Jul 16 22:34:47.897602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.897565    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerStarted","Data":"e005d4cc7f1f1b57176bf1426d229094d42c261222109c9d1999878ac7410e02"}
Jul 16 22:34:47.898034 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.897624    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerStarted","Data":"dadf4c5567900a14ce6f319948fca823a74c435a48bba24bad224c05d928985e"}
Jul 16 22:34:47.899322 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.899300    2581 scope.go:117] "RemoveContainer" containerID="038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f"
Jul 16 22:34:47.899547 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:47.899527    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz_odh-ai-gateway-infra(63157b2b-a51f-44ff-a005-5288b504d94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" podUID="63157b2b-a51f-44ff-a005-5288b504d94f"
Jul 16 22:34:47.916658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:47.916618    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" podStartSLOduration=0.916604639 podStartE2EDuration="916.604639ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:34:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:34:47.914218726 +0000 UTC m=+2875.243658630" watchObservedRunningTime="2026-07-16 22:34:47.916604639 +0000 UTC m=+2875.246044599"
Jul 16 22:34:48.905525 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:48.905490    2581 generic.go:356] "Generic (PLEG): container finished" podID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerID="e005d4cc7f1f1b57176bf1426d229094d42c261222109c9d1999878ac7410e02" exitCode=6
Jul 16 22:34:48.906021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:48.905532    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerDied","Data":"e005d4cc7f1f1b57176bf1426d229094d42c261222109c9d1999878ac7410e02"}
Jul 16 22:34:48.906021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:48.905765    2581 scope.go:117] "RemoveContainer" containerID="e005d4cc7f1f1b57176bf1426d229094d42c261222109c9d1999878ac7410e02"
Jul 16 22:34:49.912919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:49.912885    2581 generic.go:356] "Generic (PLEG): container finished" podID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerID="ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810" exitCode=6
Jul 16 22:34:49.913377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:49.912963    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerDied","Data":"ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810"}
Jul 16 22:34:49.913377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:49.913029    2581 scope.go:117] "RemoveContainer" containerID="e005d4cc7f1f1b57176bf1426d229094d42c261222109c9d1999878ac7410e02"
Jul 16 22:34:49.913377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:49.913260    2581 scope.go:117] "RemoveContainer" containerID="ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810"
Jul 16 22:34:49.913507 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:49.913477    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k_odh-ai-gateway-infra(d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2"
Jul 16 22:34:50.919558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:50.919532    2581 scope.go:117] "RemoveContainer" containerID="ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810"
Jul 16 22:34:50.919943 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:50.919689    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k_odh-ai-gateway-infra(d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2"
Jul 16 22:34:59.155370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:59.155329    2581 scope.go:117] "RemoveContainer" containerID="038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f"
Jul 16 22:34:59.962035 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:59.961996    2581 generic.go:356] "Generic (PLEG): container finished" podID="63157b2b-a51f-44ff-a005-5288b504d94f" containerID="e7817bf2b16845918b240265aac54c6146f72db1dd5abb800c52d4551cef129f" exitCode=6
Jul 16 22:34:59.962197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:59.962069    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" event={"ID":"63157b2b-a51f-44ff-a005-5288b504d94f","Type":"ContainerDied","Data":"e7817bf2b16845918b240265aac54c6146f72db1dd5abb800c52d4551cef129f"}
Jul 16 22:34:59.962197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:59.962164    2581 scope.go:117] "RemoveContainer" containerID="038ec4ebe4858653e2477aaa59638daa63467ddb4c175c18dd4cb0bbdddb289f"
Jul 16 22:34:59.962483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:34:59.962468    2581 scope.go:117] "RemoveContainer" containerID="e7817bf2b16845918b240265aac54c6146f72db1dd5abb800c52d4551cef129f"
Jul 16 22:34:59.962711 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:34:59.962690    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz_odh-ai-gateway-infra(63157b2b-a51f-44ff-a005-5288b504d94f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" podUID="63157b2b-a51f-44ff-a005-5288b504d94f"
Jul 16 22:35:00.989682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:00.989646    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:35:01.125427 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.125396    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:35:01.209052 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.209008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca\") pod \"63157b2b-a51f-44ff-a005-5288b504d94f\" (UID: \"63157b2b-a51f-44ff-a005-5288b504d94f\") "
Jul 16 22:35:01.209413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.209389    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63157b2b-a51f-44ff-a005-5288b504d94f" (UID: "63157b2b-a51f-44ff-a005-5288b504d94f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:01.310179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.310082    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63157b2b-a51f-44ff-a005-5288b504d94f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:01.974474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.974442    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"
Jul 16 22:35:01.974474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.974452    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz" event={"ID":"63157b2b-a51f-44ff-a005-5288b504d94f","Type":"ContainerDied","Data":"1cf823399158ae805e93be7d6b2fd6f0546cc16150ec0de738089f3b122af430"}
Jul 16 22:35:01.974668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.974510    2581 scope.go:117] "RemoveContainer" containerID="e7817bf2b16845918b240265aac54c6146f72db1dd5abb800c52d4551cef129f"
Jul 16 22:35:01.995954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:01.995928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:35:02.002657 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.002632    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-s7gtz"]
Jul 16 22:35:02.155165 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.155134    2581 scope.go:117] "RemoveContainer" containerID="ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810"
Jul 16 22:35:02.421919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.421880    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:35:02.980471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.980437    2581 generic.go:356] "Generic (PLEG): container finished" podID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerID="ce03eaa18642bf2eef17ebdfb363dec11dcc05c8cce855dd0be70a71e68d5b80" exitCode=6
Jul 16 22:35:02.980642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.980506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerDied","Data":"ce03eaa18642bf2eef17ebdfb363dec11dcc05c8cce855dd0be70a71e68d5b80"}
Jul 16 22:35:02.980642 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.980562    2581 scope.go:117] "RemoveContainer" containerID="ed9811d73b1cd1ad59833fd049d923e5e14dbe7cf5a88eba121654fa39636810"
Jul 16 22:35:02.980935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:02.980907    2581 scope.go:117] "RemoveContainer" containerID="ce03eaa18642bf2eef17ebdfb363dec11dcc05c8cce855dd0be70a71e68d5b80"
Jul 16 22:35:02.981161 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:02.981139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k_odh-ai-gateway-infra(d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2"
Jul 16 22:35:03.161330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.161295    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" path="/var/lib/kubelet/pods/63157b2b-a51f-44ff-a005-5288b504d94f/volumes"
Jul 16 22:35:03.171736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.171706    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:03.172066 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172054    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:35:03.172111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172068    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c56090f5-34ca-41e5-8a66-4d4ee2aa1188" containerName="revoke-keys"
Jul 16 22:35:03.172111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172083    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172088    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172096    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172101    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172286 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172161    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172286 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172172    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.172286 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.172178    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:03.176500 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.176480    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:03.193656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.193624    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:03.226831 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.226787    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-sj2zn\" (UID: \"e0bfaab5-33df-4db0-a8fa-49dffad65a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:03.327375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.327326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-sj2zn\" (UID: \"e0bfaab5-33df-4db0-a8fa-49dffad65a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:03.327936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.327918    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-sj2zn\" (UID: \"e0bfaab5-33df-4db0-a8fa-49dffad65a29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:03.486115 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.486012    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:03.646546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.646519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:03.649317 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:35:03.649282    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0bfaab5_33df_4db0_a8fa_49dffad65a29.slice/crio-26817e1fb24ceccbcb290f08d0f86c50ce117ed9000649d7b57c1575baed8e79 WatchSource:0}: Error finding container 26817e1fb24ceccbcb290f08d0f86c50ce117ed9000649d7b57c1575baed8e79: Status 404 returned error can't find the container with id 26817e1fb24ceccbcb290f08d0f86c50ce117ed9000649d7b57c1575baed8e79
Jul 16 22:35:03.989965 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.989927    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerStarted","Data":"6c48703fd3ee465617dfe3677326d067faa6d5242b09dc93d82cdcf3584d982c"}
Jul 16 22:35:03.990168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:03.989990    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerStarted","Data":"26817e1fb24ceccbcb290f08d0f86c50ce117ed9000649d7b57c1575baed8e79"}
Jul 16 22:35:04.016199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.016163    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:35:04.061819 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.061755    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" podStartSLOduration=1.061736566 podStartE2EDuration="1.061736566s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:04.035680759 +0000 UTC m=+2891.365120661" watchObservedRunningTime="2026-07-16 22:35:04.061736566 +0000 UTC m=+2891.391176514"
Jul 16 22:35:04.164535 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.164500    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:35:04.246742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.246710    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca\") pod \"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2\" (UID: \"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2\") "
Jul 16 22:35:04.247149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.247119    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" (UID: "d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:04.348008 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.347973    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:04.996794 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.996754    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerID="6c48703fd3ee465617dfe3677326d067faa6d5242b09dc93d82cdcf3584d982c" exitCode=6
Jul 16 22:35:04.997009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.996853    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerDied","Data":"6c48703fd3ee465617dfe3677326d067faa6d5242b09dc93d82cdcf3584d982c"}
Jul 16 22:35:04.997062 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.997025    2581 scope.go:117] "RemoveContainer" containerID="6c48703fd3ee465617dfe3677326d067faa6d5242b09dc93d82cdcf3584d982c"
Jul 16 22:35:04.998314 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.998290    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"
Jul 16 22:35:04.998424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.998295    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k" event={"ID":"d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2","Type":"ContainerDied","Data":"dadf4c5567900a14ce6f319948fca823a74c435a48bba24bad224c05d928985e"}
Jul 16 22:35:04.998424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:04.998416    2581 scope.go:117] "RemoveContainer" containerID="ce03eaa18642bf2eef17ebdfb363dec11dcc05c8cce855dd0be70a71e68d5b80"
Jul 16 22:35:05.084511 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.084470    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:35:05.093388 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.093353    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mkv5k"]
Jul 16 22:35:05.160875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.160841    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" path="/var/lib/kubelet/pods/d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2/volumes"
Jul 16 22:35:05.975421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975320    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"]
Jul 16 22:35:05.975877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975811    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.975877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975829    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.975877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975856    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.975877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975864    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.975877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975876    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:05.976126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975885    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="63157b2b-a51f-44ff-a005-5288b504d94f" containerName="revoke-keys"
Jul 16 22:35:05.976126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.975989    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.976126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.976002    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:05.978982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.978957    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:05.990900 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:05.990875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"]
Jul 16 22:35:06.006189 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.006157    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerID="68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b" exitCode=6
Jul 16 22:35:06.006365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.006256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerDied","Data":"68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b"}
Jul 16 22:35:06.007437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.006784    2581 scope.go:117] "RemoveContainer" containerID="6c48703fd3ee465617dfe3677326d067faa6d5242b09dc93d82cdcf3584d982c"
Jul 16 22:35:06.007437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.006843    2581 scope.go:117] "RemoveContainer" containerID="68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b"
Jul 16 22:35:06.007437 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:06.007098    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-sj2zn_odh-ai-gateway-infra(e0bfaab5-33df-4db0-a8fa-49dffad65a29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29"
Jul 16 22:35:06.061481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.061458    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5\" (UID: \"5a42b7f6-c42c-45cf-82a3-b481364fa46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:06.162833 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.162810    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5\" (UID: \"5a42b7f6-c42c-45cf-82a3-b481364fa46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:06.163401 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.163384    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5\" (UID: \"5a42b7f6-c42c-45cf-82a3-b481364fa46e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:06.289690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.289613    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:06.421552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:06.421525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"]
Jul 16 22:35:06.423618 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:35:06.423588    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a42b7f6_c42c_45cf_82a3_b481364fa46e.slice/crio-e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d WatchSource:0}: Error finding container e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d: Status 404 returned error can't find the container with id e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d
Jul 16 22:35:07.013990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:07.013898    2581 scope.go:117] "RemoveContainer" containerID="68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b"
Jul 16 22:35:07.014409 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:07.014103    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-sj2zn_odh-ai-gateway-infra(e0bfaab5-33df-4db0-a8fa-49dffad65a29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29"
Jul 16 22:35:07.015261 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:07.015216    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5" event={"ID":"5a42b7f6-c42c-45cf-82a3-b481364fa46e","Type":"ContainerStarted","Data":"ec71dd2e3ce91a217e16aaf07768987ad23760ebe4690cc505150168601d30aa"}
Jul 16 22:35:07.015324 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:07.015286    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5" event={"ID":"5a42b7f6-c42c-45cf-82a3-b481364fa46e","Type":"ContainerStarted","Data":"e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d"}
Jul 16 22:35:07.061086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:07.061040    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5" podStartSLOduration=2.061025738 podStartE2EDuration="2.061025738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:07.060302731 +0000 UTC m=+2894.389742635" watchObservedRunningTime="2026-07-16 22:35:07.061025738 +0000 UTC m=+2894.390465640"
Jul 16 22:35:08.021611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:08.021572    2581 generic.go:356] "Generic (PLEG): container finished" podID="5a42b7f6-c42c-45cf-82a3-b481364fa46e" containerID="ec71dd2e3ce91a217e16aaf07768987ad23760ebe4690cc505150168601d30aa" exitCode=0
Jul 16 22:35:08.021991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:08.021637    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5" event={"ID":"5a42b7f6-c42c-45cf-82a3-b481364fa46e","Type":"ContainerDied","Data":"ec71dd2e3ce91a217e16aaf07768987ad23760ebe4690cc505150168601d30aa"}
Jul 16 22:35:09.156772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:09.156749    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:09.292767 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:09.292685    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca\") pod \"5a42b7f6-c42c-45cf-82a3-b481364fa46e\" (UID: \"5a42b7f6-c42c-45cf-82a3-b481364fa46e\") "
Jul 16 22:35:09.293022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:09.293000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a42b7f6-c42c-45cf-82a3-b481364fa46e" (UID: "5a42b7f6-c42c-45cf-82a3-b481364fa46e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:09.393991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:09.393961    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a42b7f6-c42c-45cf-82a3-b481364fa46e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:10.031302 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:10.031266    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5" event={"ID":"5a42b7f6-c42c-45cf-82a3-b481364fa46e","Type":"ContainerDied","Data":"e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d"}
Jul 16 22:35:10.031493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:10.031335    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e4727d6967426c1eaac347e363c98f689a7dbee7e21dedb92a7b65c0fcf3f20d"
Jul 16 22:35:10.031493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:10.031277    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5"
Jul 16 22:35:18.154878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:18.154846    2581 scope.go:117] "RemoveContainer" containerID="68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b"
Jul 16 22:35:19.073560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:19.073528    2581 generic.go:356] "Generic (PLEG): container finished" podID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerID="df8859ce064d0b14563c0e075e66c6427ff8afcff8a1463f282ed2dda0384abb" exitCode=6
Jul 16 22:35:19.073731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:19.073591    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerDied","Data":"df8859ce064d0b14563c0e075e66c6427ff8afcff8a1463f282ed2dda0384abb"}
Jul 16 22:35:19.073731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:19.073656    2581 scope.go:117] "RemoveContainer" containerID="68cb976ede59130c95b819e0751c112c28953d318aeb84181c66db70ffabac6b"
Jul 16 22:35:19.074001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:19.073982    2581 scope.go:117] "RemoveContainer" containerID="df8859ce064d0b14563c0e075e66c6427ff8afcff8a1463f282ed2dda0384abb"
Jul 16 22:35:19.074302 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:19.074275    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-sj2zn_odh-ai-gateway-infra(e0bfaab5-33df-4db0-a8fa-49dffad65a29)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29"
Jul 16 22:35:20.109829 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.109793    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:20.127256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.127203    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:35:20.247908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.247881    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:20.385082 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.385008    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca\") pod \"e0bfaab5-33df-4db0-a8fa-49dffad65a29\" (UID: \"e0bfaab5-33df-4db0-a8fa-49dffad65a29\") "
Jul 16 22:35:20.385387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.385367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0bfaab5-33df-4db0-a8fa-49dffad65a29" (UID: "e0bfaab5-33df-4db0-a8fa-49dffad65a29"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:20.485719 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:20.485686    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0bfaab5-33df-4db0-a8fa-49dffad65a29-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:21.085009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.084972    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn" event={"ID":"e0bfaab5-33df-4db0-a8fa-49dffad65a29","Type":"ContainerDied","Data":"26817e1fb24ceccbcb290f08d0f86c50ce117ed9000649d7b57c1575baed8e79"}
Jul 16 22:35:21.085173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.085012    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"
Jul 16 22:35:21.085173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.085076    2581 scope.go:117] "RemoveContainer" containerID="df8859ce064d0b14563c0e075e66c6427ff8afcff8a1463f282ed2dda0384abb"
Jul 16 22:35:21.105654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.105630    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:21.110673 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.110649    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-sj2zn"]
Jul 16 22:35:21.159545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:21.159512    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" path="/var/lib/kubelet/pods/e0bfaab5-33df-4db0-a8fa-49dffad65a29/volumes"
Jul 16 22:35:26.048071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:26.048035    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:35:34.742506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742468    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:34.742975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742958    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a42b7f6-c42c-45cf-82a3-b481364fa46e" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742976    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a42b7f6-c42c-45cf-82a3-b481364fa46e" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742984    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742989    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.742998    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743005    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743011    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743016    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743022    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743027    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743078    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743086    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743093    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d03e8e2f-24c7-41a1-b8f4-15ba73bf7bc2" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743098    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0bfaab5-33df-4db0-a8fa-49dffad65a29" containerName="revoke-keys"
Jul 16 22:35:34.743366 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.743102    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a42b7f6-c42c-45cf-82a3-b481364fa46e" containerName="revoke-keys"
Jul 16 22:35:34.745857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.745842    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:34.748592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.748570    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:35:34.748712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.748631    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:35:34.756002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.755975    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:34.909585 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:34.909544    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q\" (UID: \"88e03254-530a-4498-8ef3-6647f20f9dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:35.010404 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:35.010320    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q\" (UID: \"88e03254-530a-4498-8ef3-6647f20f9dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:35.010938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:35.010914    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q\" (UID: \"88e03254-530a-4498-8ef3-6647f20f9dda\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:35.057279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:35.057257    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:35.190360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:35.190336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:35.192722 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:35:35.192680    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88e03254_530a_4498_8ef3_6647f20f9dda.slice/crio-17693d437713e5afeb24ea4f1ea68762debcbf045b0ff5781ec8d1a26550bbc8 WatchSource:0}: Error finding container 17693d437713e5afeb24ea4f1ea68762debcbf045b0ff5781ec8d1a26550bbc8: Status 404 returned error can't find the container with id 17693d437713e5afeb24ea4f1ea68762debcbf045b0ff5781ec8d1a26550bbc8
Jul 16 22:35:36.156749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:36.156666    2581 generic.go:356] "Generic (PLEG): container finished" podID="88e03254-530a-4498-8ef3-6647f20f9dda" containerID="624ae0db87c6ad5dbdea9e4f1ff5f99732048b10052b6e919861aa09fc224c8c" exitCode=6
Jul 16 22:35:36.156749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:36.156724    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" event={"ID":"88e03254-530a-4498-8ef3-6647f20f9dda","Type":"ContainerDied","Data":"624ae0db87c6ad5dbdea9e4f1ff5f99732048b10052b6e919861aa09fc224c8c"}
Jul 16 22:35:36.157133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:36.156761    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" event={"ID":"88e03254-530a-4498-8ef3-6647f20f9dda","Type":"ContainerStarted","Data":"17693d437713e5afeb24ea4f1ea68762debcbf045b0ff5781ec8d1a26550bbc8"}
Jul 16 22:35:36.157133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:36.156945    2581 scope.go:117] "RemoveContainer" containerID="624ae0db87c6ad5dbdea9e4f1ff5f99732048b10052b6e919861aa09fc224c8c"
Jul 16 22:35:37.162637 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:37.162600    2581 generic.go:356] "Generic (PLEG): container finished" podID="88e03254-530a-4498-8ef3-6647f20f9dda" containerID="ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90" exitCode=6
Jul 16 22:35:37.163073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:37.162687    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" event={"ID":"88e03254-530a-4498-8ef3-6647f20f9dda","Type":"ContainerDied","Data":"ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90"}
Jul 16 22:35:37.163073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:37.162742    2581 scope.go:117] "RemoveContainer" containerID="624ae0db87c6ad5dbdea9e4f1ff5f99732048b10052b6e919861aa09fc224c8c"
Jul 16 22:35:37.163073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:37.162833    2581 scope.go:117] "RemoveContainer" containerID="ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90"
Jul 16 22:35:37.163073 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:37.163010    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q_odh-ai-gateway-infra(88e03254-530a-4498-8ef3-6647f20f9dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" podUID="88e03254-530a-4498-8ef3-6647f20f9dda"
Jul 16 22:35:38.168243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:38.168213    2581 scope.go:117] "RemoveContainer" containerID="ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90"
Jul 16 22:35:38.168634 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:38.168410    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q_odh-ai-gateway-infra(88e03254-530a-4498-8ef3-6647f20f9dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" podUID="88e03254-530a-4498-8ef3-6647f20f9dda"
Jul 16 22:35:41.069300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.069217    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" podUID="aa9617d3-f37f-4d7f-9c96-799677bb13da" containerName="main" containerID="cri-o://3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316" gracePeriod=45
Jul 16 22:35:41.330733 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.330672    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:35:41.361127 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361091    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361176    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dgrc\" (UniqueName: \"kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361221    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361276    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361345    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361405    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache\") pod \"aa9617d3-f37f-4d7f-9c96-799677bb13da\" (UID: \"aa9617d3-f37f-4d7f-9c96-799677bb13da\") "
Jul 16 22:35:41.361468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361443    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:35:41.361468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361453    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home" (OuterVolumeSpecName: "home") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:35:41.361685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361649    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:41.361685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361667    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:41.361812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.361688    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:35:41.364025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.363981    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:35:41.364025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.363988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc" (OuterVolumeSpecName: "kube-api-access-2dgrc") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "kube-api-access-2dgrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:35:41.364254 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.364105    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm" (OuterVolumeSpecName: "dshm") pod "aa9617d3-f37f-4d7f-9c96-799677bb13da" (UID: "aa9617d3-f37f-4d7f-9c96-799677bb13da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:35:41.462543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.462505    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa9617d3-f37f-4d7f-9c96-799677bb13da-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:41.462543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.462536    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:41.462543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.462547    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dgrc\" (UniqueName: \"kubernetes.io/projected/aa9617d3-f37f-4d7f-9c96-799677bb13da-kube-api-access-2dgrc\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:41.462841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:41.462556    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa9617d3-f37f-4d7f-9c96-799677bb13da-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:42.187627 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.187593    2581 generic.go:356] "Generic (PLEG): container finished" podID="aa9617d3-f37f-4d7f-9c96-799677bb13da" containerID="3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316" exitCode=0
Jul 16 22:35:42.188051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.187632    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"
Jul 16 22:35:42.188051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.187669    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" event={"ID":"aa9617d3-f37f-4d7f-9c96-799677bb13da","Type":"ContainerDied","Data":"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"}
Jul 16 22:35:42.188051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.187711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm" event={"ID":"aa9617d3-f37f-4d7f-9c96-799677bb13da","Type":"ContainerDied","Data":"deb427a11436cb406a232ab69cc41caa37287f9f87746ee35b753f1f4c67362e"}
Jul 16 22:35:42.188051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.187730    2581 scope.go:117] "RemoveContainer" containerID="3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"
Jul 16 22:35:42.196473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.196457    2581 scope.go:117] "RemoveContainer" containerID="3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"
Jul 16 22:35:42.196734 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:42.196714    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316\": container with ID starting with 3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316 not found: ID does not exist" containerID="3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"
Jul 16 22:35:42.196798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.196743    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316"} err="failed to get container status \"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316\": rpc error: code = NotFound desc = could not find container \"3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316\": container with ID starting with 3a15735e691a7def4214ae463ef339f26776f1f78b00602641cc1026c5866316 not found: ID does not exist"
Jul 16 22:35:42.208444 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.208420    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:35:42.211992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:42.211972    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b3fb3ce5/e2e-lifecycle-model-b3fb3ce5-kserve-66966df5ff-nwlsm"]
Jul 16 22:35:43.162078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:43.162040    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa9617d3-f37f-4d7f-9c96-799677bb13da" path="/var/lib/kubelet/pods/aa9617d3-f37f-4d7f-9c96-799677bb13da/volumes"
Jul 16 22:35:50.154710 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:50.154675    2581 scope.go:117] "RemoveContainer" containerID="ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90"
Jul 16 22:35:51.230190 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:51.230155    2581 generic.go:356] "Generic (PLEG): container finished" podID="88e03254-530a-4498-8ef3-6647f20f9dda" containerID="46dff2ec8d130670dd0914fb8a4977ffc5547cd2d4f1cd426834e64133e26dc3" exitCode=6
Jul 16 22:35:51.230647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:51.230250    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" event={"ID":"88e03254-530a-4498-8ef3-6647f20f9dda","Type":"ContainerDied","Data":"46dff2ec8d130670dd0914fb8a4977ffc5547cd2d4f1cd426834e64133e26dc3"}
Jul 16 22:35:51.230647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:51.230314    2581 scope.go:117] "RemoveContainer" containerID="ae6a7452f7ce7f6bc270167c9a016254f77e6c6202672bb2aac96642b5cc0d90"
Jul 16 22:35:51.230647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:51.230564    2581 scope.go:117] "RemoveContainer" containerID="46dff2ec8d130670dd0914fb8a4977ffc5547cd2d4f1cd426834e64133e26dc3"
Jul 16 22:35:51.230834 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:51.230780    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q_odh-ai-gateway-infra(88e03254-530a-4498-8ef3-6647f20f9dda)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" podUID="88e03254-530a-4498-8ef3-6647f20f9dda"
Jul 16 22:35:52.257513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:52.257484    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:52.390436 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:52.390408    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:52.456175 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:52.456146    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca\") pod \"88e03254-530a-4498-8ef3-6647f20f9dda\" (UID: \"88e03254-530a-4498-8ef3-6647f20f9dda\") "
Jul 16 22:35:52.456492 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:52.456471    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "88e03254-530a-4498-8ef3-6647f20f9dda" (UID: "88e03254-530a-4498-8ef3-6647f20f9dda"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:52.557440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:52.557372    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/88e03254-530a-4498-8ef3-6647f20f9dda-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:35:53.201844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.201814    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:35:53.202131 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202120    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202133    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202143    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202149    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202163    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202169    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa9617d3-f37f-4d7f-9c96-799677bb13da" containerName="main"
Jul 16 22:35:53.202176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202174    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa9617d3-f37f-4d7f-9c96-799677bb13da" containerName="main"
Jul 16 22:35:53.202405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202247    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202258    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:35:53.202405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.202263    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa9617d3-f37f-4d7f-9c96-799677bb13da" containerName="main"
Jul 16 22:35:53.205047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.205030    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:35:53.214550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.214525    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:35:53.242861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.242835    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q" event={"ID":"88e03254-530a-4498-8ef3-6647f20f9dda","Type":"ContainerDied","Data":"17693d437713e5afeb24ea4f1ea68762debcbf045b0ff5781ec8d1a26550bbc8"}
Jul 16 22:35:53.242993 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.242872    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"
Jul 16 22:35:53.242993 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.242890    2581 scope.go:117] "RemoveContainer" containerID="46dff2ec8d130670dd0914fb8a4977ffc5547cd2d4f1cd426834e64133e26dc3"
Jul 16 22:35:53.261485 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.261464    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:53.263037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.263013    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-k2jwd\" (UID: \"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:35:53.267715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.267697    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-4rw4q"]
Jul 16 22:35:53.364406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.364371    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-k2jwd\" (UID: \"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:35:53.364965 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.364947    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-k2jwd\" (UID: \"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:35:53.516514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.516434    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:35:53.653737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:53.653716    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:35:53.655271 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:35:53.655240    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b7b1fa_dae2_47c7_bc62_a32af6812f1f.slice/crio-f506541760d3d8805bf973b272fc76788457279fe6a8504af68c732578238a63 WatchSource:0}: Error finding container f506541760d3d8805bf973b272fc76788457279fe6a8504af68c732578238a63: Status 404 returned error can't find the container with id f506541760d3d8805bf973b272fc76788457279fe6a8504af68c732578238a63
Jul 16 22:35:54.256569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:54.256534    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerStarted","Data":"f64b8cacb68f19789da8119709011b05e065a30ff2d2f22d33da435cadff319a"}
Jul 16 22:35:54.256720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:54.256622    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerStarted","Data":"f506541760d3d8805bf973b272fc76788457279fe6a8504af68c732578238a63"}
Jul 16 22:35:54.276533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:54.276491    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" podStartSLOduration=1.276478889 podStartE2EDuration="1.276478889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:54.275401056 +0000 UTC m=+2941.604840960" watchObservedRunningTime="2026-07-16 22:35:54.276478889 +0000 UTC m=+2941.605918791"
Jul 16 22:35:55.160243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:55.160205    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" path="/var/lib/kubelet/pods/88e03254-530a-4498-8ef3-6647f20f9dda/volumes"
Jul 16 22:35:55.263411 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:55.263375    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerID="f64b8cacb68f19789da8119709011b05e065a30ff2d2f22d33da435cadff319a" exitCode=6
Jul 16 22:35:55.263576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:55.263417    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerDied","Data":"f64b8cacb68f19789da8119709011b05e065a30ff2d2f22d33da435cadff319a"}
Jul 16 22:35:55.263641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:55.263626    2581 scope.go:117] "RemoveContainer" containerID="f64b8cacb68f19789da8119709011b05e065a30ff2d2f22d33da435cadff319a"
Jul 16 22:35:56.270361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:56.270264    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerID="49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d" exitCode=6
Jul 16 22:35:56.270361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:56.270345    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerDied","Data":"49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d"}
Jul 16 22:35:56.270800 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:56.270404    2581 scope.go:117] "RemoveContainer" containerID="f64b8cacb68f19789da8119709011b05e065a30ff2d2f22d33da435cadff319a"
Jul 16 22:35:56.270800 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:56.270531    2581 scope.go:117] "RemoveContainer" containerID="49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d"
Jul 16 22:35:56.270800 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:56.270770    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-k2jwd_odh-ai-gateway-infra(b9b7b1fa-dae2-47c7-bc62-a32af6812f1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f"
Jul 16 22:35:57.276547 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:35:57.276522    2581 scope.go:117] "RemoveContainer" containerID="49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d"
Jul 16 22:35:57.276917 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:35:57.276710    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-k2jwd_odh-ai-gateway-infra(b9b7b1fa-dae2-47c7-bc62-a32af6812f1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f"
Jul 16 22:36:08.155744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:08.155709    2581 scope.go:117] "RemoveContainer" containerID="49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d"
Jul 16 22:36:09.331720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:09.331686    2581 generic.go:356] "Generic (PLEG): container finished" podID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerID="f57c8c944e48f200917967ca0886c75d631cfa9b29f6e09abe9f728230872ceb" exitCode=6
Jul 16 22:36:09.332123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:09.331760    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerDied","Data":"f57c8c944e48f200917967ca0886c75d631cfa9b29f6e09abe9f728230872ceb"}
Jul 16 22:36:09.332123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:09.331825    2581 scope.go:117] "RemoveContainer" containerID="49533ff7ceeab35511ebe9b163c6ef8065756833eb1ff09c6422b77f8e445c1d"
Jul 16 22:36:09.332123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:09.332120    2581 scope.go:117] "RemoveContainer" containerID="f57c8c944e48f200917967ca0886c75d631cfa9b29f6e09abe9f728230872ceb"
Jul 16 22:36:09.332388 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:36:09.332366    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-k2jwd_odh-ai-gateway-infra(b9b7b1fa-dae2-47c7-bc62-a32af6812f1f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f"
Jul 16 22:36:10.357835 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:10.357797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:36:10.494407 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:10.494383    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:36:10.611406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:10.611302    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca\") pod \"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f\" (UID: \"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f\") "
Jul 16 22:36:10.611632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:10.611610    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" (UID: "b9b7b1fa-dae2-47c7-bc62-a32af6812f1f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:10.712682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:10.712649    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:11.342828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:11.342797    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"
Jul 16 22:36:11.342978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:11.342827    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd" event={"ID":"b9b7b1fa-dae2-47c7-bc62-a32af6812f1f","Type":"ContainerDied","Data":"f506541760d3d8805bf973b272fc76788457279fe6a8504af68c732578238a63"}
Jul 16 22:36:11.342978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:11.342892    2581 scope.go:117] "RemoveContainer" containerID="f57c8c944e48f200917967ca0886c75d631cfa9b29f6e09abe9f728230872ceb"
Jul 16 22:36:11.365015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:11.364987    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:36:11.372095 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:11.372072    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-k2jwd"]
Jul 16 22:36:13.161026 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:13.160991    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" path="/var/lib/kubelet/pods/b9b7b1fa-dae2-47c7-bc62-a32af6812f1f/volumes"
Jul 16 22:36:33.396648 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:33.396610    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:36:33.397160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:33.396879    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" containerID="cri-o://75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a" gracePeriod=30
Jul 16 22:36:36.784463 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:36.784424    2581 patch_prober.go:28] interesting pod/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:36:36.784899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:36.784491    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:36:37.525331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:37.525294    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:36:38.649471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.649449    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:36:38.728712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728644    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728676    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728697    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728729    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728857    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728923    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728960    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.728991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728990    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.729332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.728988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:38.729332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729037    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data\") pod \"f4de4181-ff14-4c1f-b673-b13debcf2fef\" (UID: \"f4de4181-ff14-4c1f-b673-b13debcf2fef\") "
Jul 16 22:36:38.729332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729195    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:38.729332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729206    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:38.729558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729384    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.729558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729404    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.729558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729420    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.729704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.729685    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:38.730083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.730061    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data" (OuterVolumeSpecName: "istio-data") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:38.731089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.731060    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token" (OuterVolumeSpecName: "istio-token") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:38.731089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.731080    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:38.731508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.731489    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:36:38.731578 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.731505    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x" (OuterVolumeSpecName: "kube-api-access-rrd4x") pod "f4de4181-ff14-4c1f-b673-b13debcf2fef" (UID: "f4de4181-ff14-4c1f-b673-b13debcf2fef"). InnerVolumeSpecName "kube-api-access-rrd4x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:38.830301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830269    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f4de4181-ff14-4c1f-b673-b13debcf2fef-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.830301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830294    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.830301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830303    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.830513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830312    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.830513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830320    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f4de4181-ff14-4c1f-b673-b13debcf2fef-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:38.830513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:38.830328    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrd4x\" (UniqueName: \"kubernetes.io/projected/f4de4181-ff14-4c1f-b673-b13debcf2fef-kube-api-access-rrd4x\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:36:39.467997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.467911    2581 generic.go:356] "Generic (PLEG): container finished" podID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerID="75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a" exitCode=0
Jul 16 22:36:39.467997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.467952    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"
Jul 16 22:36:39.467997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.467967    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerDied","Data":"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"}
Jul 16 22:36:39.468210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.468020    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz" event={"ID":"f4de4181-ff14-4c1f-b673-b13debcf2fef","Type":"ContainerDied","Data":"c95d87a506de83c326d664d42e18cf013262d76e6c9da91726d76f1864046917"}
Jul 16 22:36:39.468210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.468045    2581 scope.go:117] "RemoveContainer" containerID="75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"
Jul 16 22:36:39.476377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.476360    2581 scope.go:117] "RemoveContainer" containerID="83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"
Jul 16 22:36:39.483661 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.483643    2581 scope.go:117] "RemoveContainer" containerID="75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"
Jul 16 22:36:39.483940 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:36:39.483922    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a\": container with ID starting with 75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a not found: ID does not exist" containerID="75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"
Jul 16 22:36:39.483983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.483948    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a"} err="failed to get container status \"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a\": rpc error: code = NotFound desc = could not find container \"75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a\": container with ID starting with 75dc97798cc8bce8406833bf677dcc556d90ac6e7965b1337f9167b4298bd37a not found: ID does not exist"
Jul 16 22:36:39.483983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.483965    2581 scope.go:117] "RemoveContainer" containerID="83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"
Jul 16 22:36:39.484172 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:36:39.484160    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2\": container with ID starting with 83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2 not found: ID does not exist" containerID="83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"
Jul 16 22:36:39.484209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.484175    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2"} err="failed to get container status \"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2\": rpc error: code = NotFound desc = could not find container \"83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2\": container with ID starting with 83a9ee11b263c453e2168e8bda970968f8a9f6c78fb26b6cf3ab41eef57f86a2 not found: ID does not exist"
Jul 16 22:36:39.526883 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.526855    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:36:39.552910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:39.552879    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-b3fb3ce5-openshift-default-c9c69f887-vw5lz"]
Jul 16 22:36:41.160456 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:41.160422    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" path="/var/lib/kubelet/pods/f4de4181-ff14-4c1f-b673-b13debcf2fef/volumes"
Jul 16 22:36:43.318588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318554    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318872    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318886    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318894    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318914    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.318990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.318922    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319013    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319023    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319029    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="88e03254-530a-4498-8ef3-6647f20f9dda" containerName="revoke-keys"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319036    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319050    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:36:43.319306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.319061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:36:43.323632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.323614    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:36:43.326858 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.326838    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:36:43.327022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.326999    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:36:43.360133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.360098    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:36:43.368814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.368790    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-bzw97\" (UID: \"1611ea3c-95eb-448a-93db-4dbe69f3cf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:36:43.469580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.469546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-bzw97\" (UID: \"1611ea3c-95eb-448a-93db-4dbe69f3cf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:36:43.470181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.470162    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-bzw97\" (UID: \"1611ea3c-95eb-448a-93db-4dbe69f3cf2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:36:43.633675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.633643    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:36:43.767046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:43.767021    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:36:43.769160 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:36:43.769133    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1611ea3c_95eb_448a_93db_4dbe69f3cf2e.slice/crio-001c0ccf2a1b53facefde8892ad51c146c4750a6c74c27d6444047f9f4b0e949 WatchSource:0}: Error finding container 001c0ccf2a1b53facefde8892ad51c146c4750a6c74c27d6444047f9f4b0e949: Status 404 returned error can't find the container with id 001c0ccf2a1b53facefde8892ad51c146c4750a6c74c27d6444047f9f4b0e949
Jul 16 22:36:44.492365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:44.492327    2581 generic.go:356] "Generic (PLEG): container finished" podID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerID="2c70a68f85384b5271421a421cbbcf4a8554091ed10e7104c74c8638ca4520ba" exitCode=6
Jul 16 22:36:44.492709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:44.492418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerDied","Data":"2c70a68f85384b5271421a421cbbcf4a8554091ed10e7104c74c8638ca4520ba"}
Jul 16 22:36:44.492709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:44.492481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerStarted","Data":"001c0ccf2a1b53facefde8892ad51c146c4750a6c74c27d6444047f9f4b0e949"}
Jul 16 22:36:44.492709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:44.492546    2581 scope.go:117] "RemoveContainer" containerID="2c70a68f85384b5271421a421cbbcf4a8554091ed10e7104c74c8638ca4520ba"
Jul 16 22:36:45.498306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:45.498273    2581 generic.go:356] "Generic (PLEG): container finished" podID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerID="6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce" exitCode=6
Jul 16 22:36:45.498746 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:45.498370    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerDied","Data":"6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce"}
Jul 16 22:36:45.498746 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:45.498432    2581 scope.go:117] "RemoveContainer" containerID="2c70a68f85384b5271421a421cbbcf4a8554091ed10e7104c74c8638ca4520ba"
Jul 16 22:36:45.498746 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:45.498632    2581 scope.go:117] "RemoveContainer" containerID="6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce"
Jul 16 22:36:45.498880 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:36:45.498845    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-bzw97_odh-ai-gateway-infra(1611ea3c-95eb-448a-93db-4dbe69f3cf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e"
Jul 16 22:36:46.504430 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:46.504399    2581 scope.go:117] "RemoveContainer" containerID="6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce"
Jul 16 22:36:46.504842 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:36:46.504582    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-bzw97_odh-ai-gateway-infra(1611ea3c-95eb-448a-93db-4dbe69f3cf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e"
Jul 16 22:36:50.731683 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:50.731651    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:36:54.512046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:54.512020    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:36:54.541775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:36:54.541747    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:37:02.155424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:02.155386    2581 scope.go:117] "RemoveContainer" containerID="6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce"
Jul 16 22:37:02.577764 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:02.577672    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerStarted","Data":"53790b4b78f387c98cb8d67d350c9702d273725fcea9e61af44410da048d3a1b"}
Jul 16 22:37:02.597019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:02.596968    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" podStartSLOduration=19.59695401 podStartE2EDuration="19.59695401s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:02.594591791 +0000 UTC m=+3009.924031694" watchObservedRunningTime="2026-07-16 22:37:02.59695401 +0000 UTC m=+3009.926393911"
Jul 16 22:37:03.583707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:03.583668    2581 generic.go:356] "Generic (PLEG): container finished" podID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerID="53790b4b78f387c98cb8d67d350c9702d273725fcea9e61af44410da048d3a1b" exitCode=6
Jul 16 22:37:03.584164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:03.583719    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerDied","Data":"53790b4b78f387c98cb8d67d350c9702d273725fcea9e61af44410da048d3a1b"}
Jul 16 22:37:03.584164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:03.583768    2581 scope.go:117] "RemoveContainer" containerID="6e2fb9334ca098b86d61016aedffddf79947db629fffd9bfd40340a8d43276ce"
Jul 16 22:37:03.584164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:03.584016    2581 scope.go:117] "RemoveContainer" containerID="53790b4b78f387c98cb8d67d350c9702d273725fcea9e61af44410da048d3a1b"
Jul 16 22:37:03.584374 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:03.584208    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-bzw97_odh-ai-gateway-infra(1611ea3c-95eb-448a-93db-4dbe69f3cf2e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e"
Jul 16 22:37:03.607799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:03.607772    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:37:04.722257 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:04.722215    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:37:04.855586 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:04.855498    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca\") pod \"1611ea3c-95eb-448a-93db-4dbe69f3cf2e\" (UID: \"1611ea3c-95eb-448a-93db-4dbe69f3cf2e\") "
Jul 16 22:37:04.855940 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:04.855916    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1611ea3c-95eb-448a-93db-4dbe69f3cf2e" (UID: "1611ea3c-95eb-448a-93db-4dbe69f3cf2e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:04.957019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:04.956980    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1611ea3c-95eb-448a-93db-4dbe69f3cf2e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:37:05.594717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.594679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97" event={"ID":"1611ea3c-95eb-448a-93db-4dbe69f3cf2e","Type":"ContainerDied","Data":"001c0ccf2a1b53facefde8892ad51c146c4750a6c74c27d6444047f9f4b0e949"}
Jul 16 22:37:05.594717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.594717    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"
Jul 16 22:37:05.594944 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.594755    2581 scope.go:117] "RemoveContainer" containerID="53790b4b78f387c98cb8d67d350c9702d273725fcea9e61af44410da048d3a1b"
Jul 16 22:37:05.641747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.641706    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:37:05.667729 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.667700    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-bzw97"]
Jul 16 22:37:05.733030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.732994    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733355    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b9b7b1fa-dae2-47c7-bc62-a32af6812f1f" containerName="revoke-keys"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733364    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733370    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f4de4181-ff14-4c1f-b673-b13debcf2fef" containerName="istio-proxy"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733383    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733388    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733397    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733402    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733411    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733416    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733461    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733467    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.733654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.733475    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" containerName="revoke-keys"
Jul 16 22:37:05.737662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.737645    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:05.740718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.740696    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:37:05.741075 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.741059    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:37:05.750868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.750845    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:05.865821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.863916    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf\" (UID: \"6a62a468-9ca1-4ae5-90fd-92d8875cf015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:05.965399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.965355    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf\" (UID: \"6a62a468-9ca1-4ae5-90fd-92d8875cf015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:05.965992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:05.965972    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf\" (UID: \"6a62a468-9ca1-4ae5-90fd-92d8875cf015\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:06.047698 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:06.047662    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:06.204243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:06.204204    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:06.205023 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:06.205001    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a62a468_9ca1_4ae5_90fd_92d8875cf015.slice/crio-b9c6dbb4052989254d07ba79c10b4b3d72f6ea6b81019f692f823fbb31965f15 WatchSource:0}: Error finding container b9c6dbb4052989254d07ba79c10b4b3d72f6ea6b81019f692f823fbb31965f15: Status 404 returned error can't find the container with id b9c6dbb4052989254d07ba79c10b4b3d72f6ea6b81019f692f823fbb31965f15
Jul 16 22:37:06.600955 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:06.600918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerStarted","Data":"efb02fcbe4d92eb0abcf287e7df2ff12ac8fddb0afda9ce7afc16eec9f7d7360"}
Jul 16 22:37:06.601150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:06.600975    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerStarted","Data":"b9c6dbb4052989254d07ba79c10b4b3d72f6ea6b81019f692f823fbb31965f15"}
Jul 16 22:37:06.619546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:06.619499    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" podStartSLOduration=1.619485133 podStartE2EDuration="1.619485133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:06.618404125 +0000 UTC m=+3013.947844032" watchObservedRunningTime="2026-07-16 22:37:06.619485133 +0000 UTC m=+3013.948925033"
Jul 16 22:37:07.159857 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:07.159760    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1611ea3c-95eb-448a-93db-4dbe69f3cf2e" path="/var/lib/kubelet/pods/1611ea3c-95eb-448a-93db-4dbe69f3cf2e/volumes"
Jul 16 22:37:07.606581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:07.606548    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerID="efb02fcbe4d92eb0abcf287e7df2ff12ac8fddb0afda9ce7afc16eec9f7d7360" exitCode=6
Jul 16 22:37:07.606761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:07.606597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerDied","Data":"efb02fcbe4d92eb0abcf287e7df2ff12ac8fddb0afda9ce7afc16eec9f7d7360"}
Jul 16 22:37:07.606807 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:07.606778    2581 scope.go:117] "RemoveContainer" containerID="efb02fcbe4d92eb0abcf287e7df2ff12ac8fddb0afda9ce7afc16eec9f7d7360"
Jul 16 22:37:08.611902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:08.611864    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerID="bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48" exitCode=6
Jul 16 22:37:08.612381 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:08.611952    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerDied","Data":"bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48"}
Jul 16 22:37:08.612381 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:08.612010    2581 scope.go:117] "RemoveContainer" containerID="efb02fcbe4d92eb0abcf287e7df2ff12ac8fddb0afda9ce7afc16eec9f7d7360"
Jul 16 22:37:08.612381 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:08.612128    2581 scope.go:117] "RemoveContainer" containerID="bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48"
Jul 16 22:37:08.612381 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:08.612367    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf_odh-ai-gateway-infra(6a62a468-9ca1-4ae5-90fd-92d8875cf015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015"
Jul 16 22:37:09.617542 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:09.617519    2581 scope.go:117] "RemoveContainer" containerID="bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48"
Jul 16 22:37:09.617938 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:09.617741    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf_odh-ai-gateway-infra(6a62a468-9ca1-4ae5-90fd-92d8875cf015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015"
Jul 16 22:37:21.154539 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:21.154502    2581 scope.go:117] "RemoveContainer" containerID="bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48"
Jul 16 22:37:21.671299 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:21.671259    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerStarted","Data":"7d5ad795a00c4b6fe344550d2f18a4f0d568888de5d6e347fcb07f7922323ea7"}
Jul 16 22:37:22.678136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:22.678101    2581 generic.go:356] "Generic (PLEG): container finished" podID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerID="7d5ad795a00c4b6fe344550d2f18a4f0d568888de5d6e347fcb07f7922323ea7" exitCode=6
Jul 16 22:37:22.678631 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:22.678171    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerDied","Data":"7d5ad795a00c4b6fe344550d2f18a4f0d568888de5d6e347fcb07f7922323ea7"}
Jul 16 22:37:22.678631 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:22.678254    2581 scope.go:117] "RemoveContainer" containerID="bc419e3fd8a0761ae4120d1a566677d1bcdf0899cba66ce5293eb0b66f70dc48"
Jul 16 22:37:22.678631 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:22.678543    2581 scope.go:117] "RemoveContainer" containerID="7d5ad795a00c4b6fe344550d2f18a4f0d568888de5d6e347fcb07f7922323ea7"
Jul 16 22:37:22.678813 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:22.678749    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf_odh-ai-gateway-infra(6a62a468-9ca1-4ae5-90fd-92d8875cf015)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015"
Jul 16 22:37:22.696162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:22.696134    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:23.817005 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:23.816978    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:23.918708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:23.918682    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca\") pod \"6a62a468-9ca1-4ae5-90fd-92d8875cf015\" (UID: \"6a62a468-9ca1-4ae5-90fd-92d8875cf015\") "
Jul 16 22:37:23.919022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:23.919000    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a62a468-9ca1-4ae5-90fd-92d8875cf015" (UID: "6a62a468-9ca1-4ae5-90fd-92d8875cf015"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:24.020183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.020112    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a62a468-9ca1-4ae5-90fd-92d8875cf015-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:37:24.689475 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.689443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf" event={"ID":"6a62a468-9ca1-4ae5-90fd-92d8875cf015","Type":"ContainerDied","Data":"b9c6dbb4052989254d07ba79c10b4b3d72f6ea6b81019f692f823fbb31965f15"}
Jul 16 22:37:24.689475 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.689468    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"
Jul 16 22:37:24.689721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.689503    2581 scope.go:117] "RemoveContainer" containerID="7d5ad795a00c4b6fe344550d2f18a4f0d568888de5d6e347fcb07f7922323ea7"
Jul 16 22:37:24.712030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.712002    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:24.719602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.719576    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-npqbf"]
Jul 16 22:37:24.875189 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875156    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875478    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875490    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875500    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875506    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875518    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875523    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875574    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875580    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.875827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.875587    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" containerName="revoke-keys"
Jul 16 22:37:24.879543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.879526    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:24.882182 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.882156    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:37:24.882334 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.882196    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:37:24.886297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:24.886241    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:25.030293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.030191    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv\" (UID: \"7392c204-7739-4907-9634-47a262a1373e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:25.131695 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.131656    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv\" (UID: \"7392c204-7739-4907-9634-47a262a1373e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:25.132277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.132258    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv\" (UID: \"7392c204-7739-4907-9634-47a262a1373e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:25.160278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.160243    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a62a468-9ca1-4ae5-90fd-92d8875cf015" path="/var/lib/kubelet/pods/6a62a468-9ca1-4ae5-90fd-92d8875cf015/volumes"
Jul 16 22:37:25.190744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.190712    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:25.315799 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.315770    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:25.317988 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:25.317962    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7392c204_7739_4907_9634_47a262a1373e.slice/crio-4c102bb4eaa3f6b6474b02e1064354f8464bbda8fbc262dd79bdb822bea4cbf2 WatchSource:0}: Error finding container 4c102bb4eaa3f6b6474b02e1064354f8464bbda8fbc262dd79bdb822bea4cbf2: Status 404 returned error can't find the container with id 4c102bb4eaa3f6b6474b02e1064354f8464bbda8fbc262dd79bdb822bea4cbf2
Jul 16 22:37:25.695703 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.695671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerStarted","Data":"227663d71cc8947fbe9256daea60f28e9fa6e82aa5ce49a1cdb62c66d44c7e11"}
Jul 16 22:37:25.695839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.695718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerStarted","Data":"4c102bb4eaa3f6b6474b02e1064354f8464bbda8fbc262dd79bdb822bea4cbf2"}
Jul 16 22:37:25.715076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:25.715032    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" podStartSLOduration=1.7150191609999998 podStartE2EDuration="1.715019161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:25.714670539 +0000 UTC m=+3033.044110432" watchObservedRunningTime="2026-07-16 22:37:25.715019161 +0000 UTC m=+3033.044459056"
Jul 16 22:37:26.701761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:26.701719    2581 generic.go:356] "Generic (PLEG): container finished" podID="7392c204-7739-4907-9634-47a262a1373e" containerID="227663d71cc8947fbe9256daea60f28e9fa6e82aa5ce49a1cdb62c66d44c7e11" exitCode=6
Jul 16 22:37:26.702219 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:26.701794    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerDied","Data":"227663d71cc8947fbe9256daea60f28e9fa6e82aa5ce49a1cdb62c66d44c7e11"}
Jul 16 22:37:26.702219 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:26.702015    2581 scope.go:117] "RemoveContainer" containerID="227663d71cc8947fbe9256daea60f28e9fa6e82aa5ce49a1cdb62c66d44c7e11"
Jul 16 22:37:27.228360 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.228285    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:27.231686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.231669    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:27.241671 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.241633    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:27.351020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.350983    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7\" (UID: \"7011e293-1e39-485b-a777-63e2142c7f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:27.451541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.451507    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7\" (UID: \"7011e293-1e39-485b-a777-63e2142c7f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:27.452086 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.452061    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7\" (UID: \"7011e293-1e39-485b-a777-63e2142c7f9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:27.542118 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.542035    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:27.668716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.668687    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:27.670893 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:27.670865    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7011e293_1e39_485b_a777_63e2142c7f9a.slice/crio-0efd835f60f4018988233a4e1981b8a0ef112d7a8fb14dbaba1ed1f03d6d6c5f WatchSource:0}: Error finding container 0efd835f60f4018988233a4e1981b8a0ef112d7a8fb14dbaba1ed1f03d6d6c5f: Status 404 returned error can't find the container with id 0efd835f60f4018988233a4e1981b8a0ef112d7a8fb14dbaba1ed1f03d6d6c5f
Jul 16 22:37:27.707822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.707783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerStarted","Data":"0efd835f60f4018988233a4e1981b8a0ef112d7a8fb14dbaba1ed1f03d6d6c5f"}
Jul 16 22:37:27.709477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.709457    2581 generic.go:356] "Generic (PLEG): container finished" podID="7392c204-7739-4907-9634-47a262a1373e" containerID="e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4" exitCode=6
Jul 16 22:37:27.709583 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.709486    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerDied","Data":"e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4"}
Jul 16 22:37:27.709583 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.709523    2581 scope.go:117] "RemoveContainer" containerID="227663d71cc8947fbe9256daea60f28e9fa6e82aa5ce49a1cdb62c66d44c7e11"
Jul 16 22:37:27.709742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:27.709727    2581 scope.go:117] "RemoveContainer" containerID="e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4"
Jul 16 22:37:27.709998 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:27.709974    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv_odh-ai-gateway-infra(7392c204-7739-4907-9634-47a262a1373e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" podUID="7392c204-7739-4907-9634-47a262a1373e"
Jul 16 22:37:28.715770 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:28.715740    2581 generic.go:356] "Generic (PLEG): container finished" podID="7011e293-1e39-485b-a777-63e2142c7f9a" containerID="a41aac4e0cede1a9ed518bb2a1c13cee2413f3a86a886fc162a6c87f64edd7aa" exitCode=6
Jul 16 22:37:28.716210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:28.715837    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerDied","Data":"a41aac4e0cede1a9ed518bb2a1c13cee2413f3a86a886fc162a6c87f64edd7aa"}
Jul 16 22:37:28.716210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:28.715991    2581 scope.go:117] "RemoveContainer" containerID="a41aac4e0cede1a9ed518bb2a1c13cee2413f3a86a886fc162a6c87f64edd7aa"
Jul 16 22:37:28.717525 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:28.717508    2581 scope.go:117] "RemoveContainer" containerID="e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4"
Jul 16 22:37:28.717680 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:28.717665    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv_odh-ai-gateway-infra(7392c204-7739-4907-9634-47a262a1373e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" podUID="7392c204-7739-4907-9634-47a262a1373e"
Jul 16 22:37:29.723437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:29.723349    2581 generic.go:356] "Generic (PLEG): container finished" podID="7011e293-1e39-485b-a777-63e2142c7f9a" containerID="360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c" exitCode=6
Jul 16 22:37:29.723898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:29.723434    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerDied","Data":"360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c"}
Jul 16 22:37:29.723898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:29.723494    2581 scope.go:117] "RemoveContainer" containerID="a41aac4e0cede1a9ed518bb2a1c13cee2413f3a86a886fc162a6c87f64edd7aa"
Jul 16 22:37:29.723898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:29.723606    2581 scope.go:117] "RemoveContainer" containerID="360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c"
Jul 16 22:37:29.723898 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:29.723842    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7_odh-ai-gateway-infra(7011e293-1e39-485b-a777-63e2142c7f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" podUID="7011e293-1e39-485b-a777-63e2142c7f9a"
Jul 16 22:37:30.729878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:30.729853    2581 scope.go:117] "RemoveContainer" containerID="360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c"
Jul 16 22:37:30.730273 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:30.730026    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7_odh-ai-gateway-infra(7011e293-1e39-485b-a777-63e2142c7f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" podUID="7011e293-1e39-485b-a777-63e2142c7f9a"
Jul 16 22:37:31.825655 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.825619    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:37:31.830175 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.830154    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.832936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.832914    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-3a3e4b-openshift-default-dockercfg-5k78d\""
Jul 16 22:37:31.846836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.846808    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:37:31.889464 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889428    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889471    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889531    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889558    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889596    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8h7c9\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889648    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889673    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889697    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.889785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.889750    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990696 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990658    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990696 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990700    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990736    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990760    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990782    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8h7c9\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990824    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990848    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990876    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.990922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.990915    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.991264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.991191    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.991264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.991256    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.991396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.991359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.991396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.991392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.991834 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.991812    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.993514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.993494    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.993617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.993500    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.999128 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.999105    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8h7c9\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:31.999248 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:31.999207    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token\") pod \"e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") " pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:32.140620 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.140595    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:32.281151 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.281122    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:37:32.282861 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:32.282835    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2737869_15bc_42d7_ab70_e4e3fc455bfe.slice/crio-472784da2ee8280c76a83c05bebd841e8b1189cff409440d7b5c93a8c5a3a568 WatchSource:0}: Error finding container 472784da2ee8280c76a83c05bebd841e8b1189cff409440d7b5c93a8c5a3a568: Status 404 returned error can't find the container with id 472784da2ee8280c76a83c05bebd841e8b1189cff409440d7b5c93a8c5a3a568
Jul 16 22:37:32.285143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.285111    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:32.285269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.285186    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:32.285269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.285249    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:32.742199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.742163    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerStarted","Data":"3358d700765e754baa7936d842092324784413c99996d989152605bdf760810a"}
Jul 16 22:37:32.742377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.742219    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerStarted","Data":"472784da2ee8280c76a83c05bebd841e8b1189cff409440d7b5c93a8c5a3a568"}
Jul 16 22:37:32.768129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:32.768076    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podStartSLOduration=1.768058363 podStartE2EDuration="1.768058363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:32.76436219 +0000 UTC m=+3040.093802091" watchObservedRunningTime="2026-07-16 22:37:32.768058363 +0000 UTC m=+3040.097498264"
Jul 16 22:37:33.140771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:33.140741    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:33.145898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:33.145870    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:33.746921 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:33.746890    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:33.748256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:33.748094    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:39.977967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:39.977933    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:37:39.981809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:39.981786    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:39.985265 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:39.985216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2fe960-openshift-default-dockercfg-cxmj5\""
Jul 16 22:37:40.021506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.021479    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:37:40.063902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.063875    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.063902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.063904    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cwh5\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.063937    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.063989    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.064026    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.064083    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.064115    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064139 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.064134    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.064480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.064205    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.155196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.155167    2581 scope.go:117] "RemoveContainer" containerID="e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4"
Jul 16 22:37:40.164981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.164951    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.164994    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165019    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165061    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165348 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165098    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165348 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165137    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cwh5\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165348 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165326    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165372    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165402    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165432    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165452    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165771 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165746    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165860    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.165966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.165940    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.168061 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.168035    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.168591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.168569    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.176645 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.176620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.177442 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.177426    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cwh5\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5\") pod \"e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") " pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.292865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.292781    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:40.438103 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.438067    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:37:40.438221 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:40.438203    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1919c46_f394_4c22_8a9e_1841ea53e13c.slice/crio-136352ef42e01e5a1cb91d4355ff95e411166cb3c21b1b2102bcf090c71065b8 WatchSource:0}: Error finding container 136352ef42e01e5a1cb91d4355ff95e411166cb3c21b1b2102bcf090c71065b8: Status 404 returned error can't find the container with id 136352ef42e01e5a1cb91d4355ff95e411166cb3c21b1b2102bcf090c71065b8
Jul 16 22:37:40.440361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.440335    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:40.440448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.440392    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:40.440448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.440420    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:40.781045 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.781007    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerStarted","Data":"94649719379d1d7c41468cf82b8d79c4cb7589c5935b59d3cb64cb531b2be882"}
Jul 16 22:37:40.781209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.781075    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerStarted","Data":"136352ef42e01e5a1cb91d4355ff95e411166cb3c21b1b2102bcf090c71065b8"}
Jul 16 22:37:40.782947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.782921    2581 generic.go:356] "Generic (PLEG): container finished" podID="7392c204-7739-4907-9634-47a262a1373e" containerID="ebead174bbc3dcb6798f6db1deb4187f89b122f936d536b5b7c1b7ef000197d6" exitCode=6
Jul 16 22:37:40.783067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.782986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerDied","Data":"ebead174bbc3dcb6798f6db1deb4187f89b122f936d536b5b7c1b7ef000197d6"}
Jul 16 22:37:40.783067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.783040    2581 scope.go:117] "RemoveContainer" containerID="e0e7cb74741ee0221d0aa3bd4cd4584ae849abdf1c84ca46a96bf4b45b558de4"
Jul 16 22:37:40.783371 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.783349    2581 scope.go:117] "RemoveContainer" containerID="ebead174bbc3dcb6798f6db1deb4187f89b122f936d536b5b7c1b7ef000197d6"
Jul 16 22:37:40.783553 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:40.783536    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv_odh-ai-gateway-infra(7392c204-7739-4907-9634-47a262a1373e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" podUID="7392c204-7739-4907-9634-47a262a1373e"
Jul 16 22:37:40.804861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:40.804824    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podStartSLOduration=1.804810604 podStartE2EDuration="1.804810604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:40.802470085 +0000 UTC m=+3048.131909988" watchObservedRunningTime="2026-07-16 22:37:40.804810604 +0000 UTC m=+3048.134250505"
Jul 16 22:37:41.293310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.293277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:41.298109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.298089    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:41.789549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.789506    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:41.790451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.790435    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:41.838478 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.838432    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:41.971869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:41.971845    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:42.082975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.082890    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca\") pod \"7392c204-7739-4907-9634-47a262a1373e\" (UID: \"7392c204-7739-4907-9634-47a262a1373e\") "
Jul 16 22:37:42.083297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.083273    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7392c204-7739-4907-9634-47a262a1373e" (UID: "7392c204-7739-4907-9634-47a262a1373e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:42.183757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.183728    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7392c204-7739-4907-9634-47a262a1373e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:37:42.795328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.795299    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"
Jul 16 22:37:42.795742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.795330    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv" event={"ID":"7392c204-7739-4907-9634-47a262a1373e","Type":"ContainerDied","Data":"4c102bb4eaa3f6b6474b02e1064354f8464bbda8fbc262dd79bdb822bea4cbf2"}
Jul 16 22:37:42.795742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.795401    2581 scope.go:117] "RemoveContainer" containerID="ebead174bbc3dcb6798f6db1deb4187f89b122f936d536b5b7c1b7ef000197d6"
Jul 16 22:37:42.816843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.816816    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:42.821508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.821485    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-zn2jv"]
Jul 16 22:37:42.943798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:42.943761    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:37:43.164104 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.164067    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7392c204-7739-4907-9634-47a262a1373e" path="/var/lib/kubelet/pods/7392c204-7739-4907-9634-47a262a1373e/volumes"
Jul 16 22:37:43.368517 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368488    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:37:43.368822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.368882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368824    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.368882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368836    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.368882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368842    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.368882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.368882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368858    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.369036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368908    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.369036 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.368919    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:37:43.373175 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.373153    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:37:43.380452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.380415    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:37:43.493049 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.492965    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-grh76\" (UID: \"b3a32371-1dd4-4588-abb8-545d1d19c937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:37:43.593597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.593536    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-grh76\" (UID: \"b3a32371-1dd4-4588-abb8-545d1d19c937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:37:43.594653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.594620    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-grh76\" (UID: \"b3a32371-1dd4-4588-abb8-545d1d19c937\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:37:43.684364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.684331    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:37:43.855257 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:43.853388    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:37:43.873626 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:37:43.873593    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a32371_1dd4_4588_abb8_545d1d19c937.slice/crio-3cdeae50b9a7687711d0419cede454d930e32b4673748bcb3e90b6aed9ec2452 WatchSource:0}: Error finding container 3cdeae50b9a7687711d0419cede454d930e32b4673748bcb3e90b6aed9ec2452: Status 404 returned error can't find the container with id 3cdeae50b9a7687711d0419cede454d930e32b4673748bcb3e90b6aed9ec2452
Jul 16 22:37:44.809759 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:44.809730    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerID="13539f0d04e3ab2b04e480f70f43c8f6f3b03513778e652687a0665773bf880e" exitCode=6
Jul 16 22:37:44.809879 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:44.809812    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" event={"ID":"b3a32371-1dd4-4588-abb8-545d1d19c937","Type":"ContainerDied","Data":"13539f0d04e3ab2b04e480f70f43c8f6f3b03513778e652687a0665773bf880e"}
Jul 16 22:37:44.809879 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:44.809862    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" event={"ID":"b3a32371-1dd4-4588-abb8-545d1d19c937","Type":"ContainerStarted","Data":"3cdeae50b9a7687711d0419cede454d930e32b4673748bcb3e90b6aed9ec2452"}
Jul 16 22:37:44.809990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:44.809976    2581 scope.go:117] "RemoveContainer" containerID="13539f0d04e3ab2b04e480f70f43c8f6f3b03513778e652687a0665773bf880e"
Jul 16 22:37:45.816460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:45.816428    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerID="0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766" exitCode=6
Jul 16 22:37:45.816814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:45.816549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" event={"ID":"b3a32371-1dd4-4588-abb8-545d1d19c937","Type":"ContainerDied","Data":"0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766"}
Jul 16 22:37:45.816814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:45.816595    2581 scope.go:117] "RemoveContainer" containerID="13539f0d04e3ab2b04e480f70f43c8f6f3b03513778e652687a0665773bf880e"
Jul 16 22:37:45.816814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:45.816775    2581 scope.go:117] "RemoveContainer" containerID="0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766"
Jul 16 22:37:45.817015 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:45.816995    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-grh76_odh-ai-gateway-infra(b3a32371-1dd4-4588-abb8-545d1d19c937)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937"
Jul 16 22:37:46.155363 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:46.155323    2581 scope.go:117] "RemoveContainer" containerID="360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c"
Jul 16 22:37:46.823424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:46.823322    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerStarted","Data":"09f99e2046898de1cbc9d9d911546d77df18339e0a85cabf1feb1280f57a9abe"}
Jul 16 22:37:46.825007 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:46.824987    2581 scope.go:117] "RemoveContainer" containerID="0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766"
Jul 16 22:37:46.825174 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:46.825155    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-grh76_odh-ai-gateway-infra(b3a32371-1dd4-4588-abb8-545d1d19c937)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937"
Jul 16 22:37:46.844292 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:46.844217    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" podStartSLOduration=19.844196951 podStartE2EDuration="19.844196951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:46.841788985 +0000 UTC m=+3054.171228889" watchObservedRunningTime="2026-07-16 22:37:46.844196951 +0000 UTC m=+3054.173636855"
Jul 16 22:37:47.831243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:47.831192    2581 generic.go:356] "Generic (PLEG): container finished" podID="7011e293-1e39-485b-a777-63e2142c7f9a" containerID="09f99e2046898de1cbc9d9d911546d77df18339e0a85cabf1feb1280f57a9abe" exitCode=6
Jul 16 22:37:47.831675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:47.831268    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerDied","Data":"09f99e2046898de1cbc9d9d911546d77df18339e0a85cabf1feb1280f57a9abe"}
Jul 16 22:37:47.831675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:47.831326    2581 scope.go:117] "RemoveContainer" containerID="360ef493e80a36553e063aad21950b54a3f2807035d9de68da6cbf3d326b1e7c"
Jul 16 22:37:47.831675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:47.831644    2581 scope.go:117] "RemoveContainer" containerID="09f99e2046898de1cbc9d9d911546d77df18339e0a85cabf1feb1280f57a9abe"
Jul 16 22:37:47.831892 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:47.831873    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7_odh-ai-gateway-infra(7011e293-1e39-485b-a777-63e2142c7f9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" podUID="7011e293-1e39-485b-a777-63e2142c7f9a"
Jul 16 22:37:47.857541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:47.857515    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:48.981277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:48.981255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:49.042017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.041985    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca\") pod \"7011e293-1e39-485b-a777-63e2142c7f9a\" (UID: \"7011e293-1e39-485b-a777-63e2142c7f9a\") "
Jul 16 22:37:49.042390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.042363    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7011e293-1e39-485b-a777-63e2142c7f9a" (UID: "7011e293-1e39-485b-a777-63e2142c7f9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:49.142772 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.142639    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7011e293-1e39-485b-a777-63e2142c7f9a-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:37:49.789452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.789409    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 16 22:37:49.789629 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.789481    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 16 22:37:49.845101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.845059    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7" event={"ID":"7011e293-1e39-485b-a777-63e2142c7f9a","Type":"ContainerDied","Data":"0efd835f60f4018988233a4e1981b8a0ef112d7a8fb14dbaba1ed1f03d6d6c5f"}
Jul 16 22:37:49.845101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.845075    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"
Jul 16 22:37:49.845377 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.845120    2581 scope.go:117] "RemoveContainer" containerID="09f99e2046898de1cbc9d9d911546d77df18339e0a85cabf1feb1280f57a9abe"
Jul 16 22:37:49.864706 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.864677    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:49.879989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:49.879963    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-dk2j7"]
Jul 16 22:37:50.854843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.854798    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerID="3358d700765e754baa7936d842092324784413c99996d989152605bdf760810a" exitCode=0
Jul 16 22:37:50.855447 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.854909    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerDied","Data":"3358d700765e754baa7936d842092324784413c99996d989152605bdf760810a"}
Jul 16 22:37:50.855529 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.855442    2581 scope.go:117] "RemoveContainer" containerID="3358d700765e754baa7936d842092324784413c99996d989152605bdf760810a"
Jul 16 22:37:50.856879 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.856838    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:50.856983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.856933    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:50.857047 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:50.856979    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:51.164246 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:51.164128    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" path="/var/lib/kubelet/pods/7011e293-1e39-485b-a777-63e2142c7f9a/volumes"
Jul 16 22:37:51.867461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:51.867422    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerStarted","Data":"b47adbfb62bc5f73b3e3b154ff09a8f3d78c11f56aa11aa94b02ced2d8d03d99"}
Jul 16 22:37:52.141333 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:52.141217    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:52.141333 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:52.141286    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Jul 16 22:37:52.141528 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:52.141340    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Jul 16 22:37:53.141598 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:53.141560    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Jul 16 22:37:53.141990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:53.141622    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Jul 16 22:37:53.929122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:53.929083    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:37:54.141151 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:54.141118    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Jul 16 22:37:54.141347 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:54.141174    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Jul 16 22:37:55.141902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:55.141861    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body=
Jul 16 22:37:55.142357 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:55.141929    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused"
Jul 16 22:37:56.145010 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:56.144977    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:56.145473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:56.145262    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:56.146061 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:56.146039    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:37:57.367944 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.367374    2581 patch_prober.go:28] interesting pod/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:56130->10.133.0.105:15021: read: connection reset by peer" start-of-body=
Jul 16 22:37:57.367944 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.367438    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": read tcp 10.133.0.2:56130->10.133.0.105:15021: read: connection reset by peer"
Jul 16 22:37:57.903397 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.903365    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerID="94649719379d1d7c41468cf82b8d79c4cb7589c5935b59d3cb64cb531b2be882" exitCode=0
Jul 16 22:37:57.903563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.903437    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerDied","Data":"94649719379d1d7c41468cf82b8d79c4cb7589c5935b59d3cb64cb531b2be882"}
Jul 16 22:37:57.903906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.903890    2581 scope.go:117] "RemoveContainer" containerID="94649719379d1d7c41468cf82b8d79c4cb7589c5935b59d3cb64cb531b2be882"
Jul 16 22:37:57.904875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.904849    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:57.904947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.904934    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:57.904983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:57.904965    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:37:58.155718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.155634    2581 scope.go:117] "RemoveContainer" containerID="0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766"
Jul 16 22:37:58.909689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.909654    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerStarted","Data":"bedbde96d81c3773a4787f0f469fb1707c4da462f2cf197a2a533b2107c3a825"}
Jul 16 22:37:58.911435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.911411    2581 generic.go:356] "Generic (PLEG): container finished" podID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerID="4344b1746941ad00d666b2ddedd7aa083e648bd57ce39e7899e33ca31bcde681" exitCode=6
Jul 16 22:37:58.911570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.911479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" event={"ID":"b3a32371-1dd4-4588-abb8-545d1d19c937","Type":"ContainerDied","Data":"4344b1746941ad00d666b2ddedd7aa083e648bd57ce39e7899e33ca31bcde681"}
Jul 16 22:37:58.911570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.911526    2581 scope.go:117] "RemoveContainer" containerID="0a919ab852b0962d249ee7e40bff1f2f96c09a1597fe1f75f5cd6e84e88f5766"
Jul 16 22:37:58.911848 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:58.911832    2581 scope.go:117] "RemoveContainer" containerID="4344b1746941ad00d666b2ddedd7aa083e648bd57ce39e7899e33ca31bcde681"
Jul 16 22:37:58.912053 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:37:58.912034    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-grh76_odh-ai-gateway-infra(b3a32371-1dd4-4588-abb8-545d1d19c937)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937"
Jul 16 22:37:59.293383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:59.293351    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:37:59.293560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:59.293507    2581 patch_prober.go:28] interesting pod/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Jul 16 22:37:59.293602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:59.293551    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Jul 16 22:37:59.962319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:37:59.962273    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:38:00.141502 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.141479    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:38:00.257595 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.257504    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca\") pod \"b3a32371-1dd4-4588-abb8-545d1d19c937\" (UID: \"b3a32371-1dd4-4588-abb8-545d1d19c937\") "
Jul 16 22:38:00.258490 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.258097    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3a32371-1dd4-4588-abb8-545d1d19c937" (UID: "b3a32371-1dd4-4588-abb8-545d1d19c937"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:00.293291 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.293262    2581 patch_prober.go:28] interesting pod/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Jul 16 22:38:00.293428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.293313    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Jul 16 22:38:00.359768 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.359728    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a32371-1dd4-4588-abb8-545d1d19c937-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:38:00.922285 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.922254    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"
Jul 16 22:38:00.922285 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.922277    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76" event={"ID":"b3a32371-1dd4-4588-abb8-545d1d19c937","Type":"ContainerDied","Data":"3cdeae50b9a7687711d0419cede454d930e32b4673748bcb3e90b6aed9ec2452"}
Jul 16 22:38:00.922553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.922336    2581 scope.go:117] "RemoveContainer" containerID="4344b1746941ad00d666b2ddedd7aa083e648bd57ce39e7899e33ca31bcde681"
Jul 16 22:38:00.963315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.963282    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:38:00.971886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:00.971857    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-grh76"]
Jul 16 22:38:01.163001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:01.162948    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" path="/var/lib/kubelet/pods/b3a32371-1dd4-4588-abb8-545d1d19c937/volumes"
Jul 16 22:38:01.293987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:01.293888    2581 patch_prober.go:28] interesting pod/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Jul 16 22:38:01.293987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:01.293959    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Jul 16 22:38:02.293978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:02.293946    2581 patch_prober.go:28] interesting pod/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body=
Jul 16 22:38:02.294384 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:02.294002    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused"
Jul 16 22:38:03.296821 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:03.296786    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:38:03.297290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:03.297163    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:38:03.297910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:03.297893    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:38:15.034179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034147    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034511    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034523    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034533    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034539    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034545    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034550    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034557    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034562    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034622    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7392c204-7739-4907-9634-47a262a1373e" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034629    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034635    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034642    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:15.034841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.034647    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:15.036484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.036469    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:15.039431 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.039405    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:38:15.039544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.039452    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:38:15.049592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.049570    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:15.188539 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.188506    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr\" (UID: \"6feed3ab-e184-40a0-8c64-ba4678d5068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:15.289470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.289367    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr\" (UID: \"6feed3ab-e184-40a0-8c64-ba4678d5068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:15.289984 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.289957    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr\" (UID: \"6feed3ab-e184-40a0-8c64-ba4678d5068f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:15.346553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.346519    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:15.470775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.470753    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:15.473088 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:38:15.473056    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6feed3ab_e184_40a0_8c64_ba4678d5068f.slice/crio-71dca83b7a1128013470b6b9866aebf0fc7814411a7e3b0c96c3f468848244e8 WatchSource:0}: Error finding container 71dca83b7a1128013470b6b9866aebf0fc7814411a7e3b0c96c3f468848244e8: Status 404 returned error can't find the container with id 71dca83b7a1128013470b6b9866aebf0fc7814411a7e3b0c96c3f468848244e8
Jul 16 22:38:15.991839 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.991802    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerStarted","Data":"010bd0e4bf28848602d2c4f074b2985deb16bfdd7231569cc9c6595a6159682a"}
Jul 16 22:38:15.992028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:15.991861    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerStarted","Data":"71dca83b7a1128013470b6b9866aebf0fc7814411a7e3b0c96c3f468848244e8"}
Jul 16 22:38:16.009054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:16.008982    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" podStartSLOduration=1.008965945 podStartE2EDuration="1.008965945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:16.007435714 +0000 UTC m=+3083.336875617" watchObservedRunningTime="2026-07-16 22:38:16.008965945 +0000 UTC m=+3083.338405846"
Jul 16 22:38:16.997818 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:16.997781    2581 generic.go:356] "Generic (PLEG): container finished" podID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerID="010bd0e4bf28848602d2c4f074b2985deb16bfdd7231569cc9c6595a6159682a" exitCode=6
Jul 16 22:38:16.998247 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:16.997870    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerDied","Data":"010bd0e4bf28848602d2c4f074b2985deb16bfdd7231569cc9c6595a6159682a"}
Jul 16 22:38:16.998247 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:16.998030    2581 scope.go:117] "RemoveContainer" containerID="010bd0e4bf28848602d2c4f074b2985deb16bfdd7231569cc9c6595a6159682a"
Jul 16 22:38:18.004935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:18.004842    2581 generic.go:356] "Generic (PLEG): container finished" podID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerID="7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115" exitCode=6
Jul 16 22:38:18.004935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:18.004890    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerDied","Data":"7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115"}
Jul 16 22:38:18.005390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:18.004952    2581 scope.go:117] "RemoveContainer" containerID="010bd0e4bf28848602d2c4f074b2985deb16bfdd7231569cc9c6595a6159682a"
Jul 16 22:38:18.005390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:18.005157    2581 scope.go:117] "RemoveContainer" containerID="7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115"
Jul 16 22:38:18.005459 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:18.005386    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr_odh-ai-gateway-infra(6feed3ab-e184-40a0-8c64-ba4678d5068f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f"
Jul 16 22:38:19.011787 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:19.011761    2581 scope.go:117] "RemoveContainer" containerID="7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115"
Jul 16 22:38:19.012174 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:19.011942    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr_odh-ai-gateway-infra(6feed3ab-e184-40a0-8c64-ba4678d5068f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f"
Jul 16 22:38:19.326459 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:19.326380    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:38:27.301904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.301869    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302186    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302197    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302222    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302244    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302295    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a32371-1dd4-4588-abb8-545d1d19c937" containerName="revoke-keys"
Jul 16 22:38:27.302315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.302306    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7011e293-1e39-485b-a777-63e2142c7f9a" containerName="revoke-keys"
Jul 16 22:38:27.304997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.304981    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:27.312598 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.312573    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:27.388960 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.388926    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2\" (UID: \"adb4aaec-4ee3-45c7-93f3-909f6c528539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:27.489580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.489546    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2\" (UID: \"adb4aaec-4ee3-45c7-93f3-909f6c528539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:27.490144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.490124    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2\" (UID: \"adb4aaec-4ee3-45c7-93f3-909f6c528539\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:27.616432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.616403    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:27.752509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:27.752483    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:27.754567 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:38:27.754544    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadb4aaec_4ee3_45c7_93f3_909f6c528539.slice/crio-1070d515cccbc816cb36189397cb018250ef57e6112d10b978b2bd730e9e1d39 WatchSource:0}: Error finding container 1070d515cccbc816cb36189397cb018250ef57e6112d10b978b2bd730e9e1d39: Status 404 returned error can't find the container with id 1070d515cccbc816cb36189397cb018250ef57e6112d10b978b2bd730e9e1d39
Jul 16 22:38:28.056106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:28.056070    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" event={"ID":"adb4aaec-4ee3-45c7-93f3-909f6c528539","Type":"ContainerStarted","Data":"1070d515cccbc816cb36189397cb018250ef57e6112d10b978b2bd730e9e1d39"}
Jul 16 22:38:29.061572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:29.061539    2581 generic.go:356] "Generic (PLEG): container finished" podID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerID="203758377b0d09335c2489df2f18a8d3a49869c051079e8f0265703b86edd06a" exitCode=6
Jul 16 22:38:29.062081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:29.061623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" event={"ID":"adb4aaec-4ee3-45c7-93f3-909f6c528539","Type":"ContainerDied","Data":"203758377b0d09335c2489df2f18a8d3a49869c051079e8f0265703b86edd06a"}
Jul 16 22:38:29.062081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:29.061849    2581 scope.go:117] "RemoveContainer" containerID="203758377b0d09335c2489df2f18a8d3a49869c051079e8f0265703b86edd06a"
Jul 16 22:38:30.068581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:30.068494    2581 generic.go:356] "Generic (PLEG): container finished" podID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerID="2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2" exitCode=6
Jul 16 22:38:30.068581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:30.068555    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" event={"ID":"adb4aaec-4ee3-45c7-93f3-909f6c528539","Type":"ContainerDied","Data":"2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2"}
Jul 16 22:38:30.069056 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:30.068604    2581 scope.go:117] "RemoveContainer" containerID="203758377b0d09335c2489df2f18a8d3a49869c051079e8f0265703b86edd06a"
Jul 16 22:38:30.069056 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:30.068751    2581 scope.go:117] "RemoveContainer" containerID="2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2"
Jul 16 22:38:30.069056 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:30.068987    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2_odh-ai-gateway-infra(adb4aaec-4ee3-45c7-93f3-909f6c528539)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539"
Jul 16 22:38:30.454638 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:30.454607    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:38:31.074440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:31.074418    2581 scope.go:117] "RemoveContainer" containerID="2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2"
Jul 16 22:38:31.074811 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:31.074608    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2_odh-ai-gateway-infra(adb4aaec-4ee3-45c7-93f3-909f6c528539)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539"
Jul 16 22:38:33.416922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.416886    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:33.419107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.419092    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:33.432151 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.432126    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:33.437930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.437905    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-jqckb\" (UID: \"36ba0928-6936-4f74-b6bd-9136788bf5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:33.538483 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.538454    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-jqckb\" (UID: \"36ba0928-6936-4f74-b6bd-9136788bf5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:33.539085 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.539065    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-jqckb\" (UID: \"36ba0928-6936-4f74-b6bd-9136788bf5c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:33.729264 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.729150    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:33.866668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:33.866539    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:34.089046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:34.089012    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" event={"ID":"36ba0928-6936-4f74-b6bd-9136788bf5c4","Type":"ContainerStarted","Data":"d83b7db72a9ec6dbf91d4a5d737af695bf93aa1796d5e5da59840440842a141b"}
Jul 16 22:38:34.155430 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:34.155408    2581 scope.go:117] "RemoveContainer" containerID="7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115"
Jul 16 22:38:35.095727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.095692    2581 generic.go:356] "Generic (PLEG): container finished" podID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerID="5afe70393c5ba8e1b8accb9d794fda4b80096468483e9f562e17521aac470c55" exitCode=6
Jul 16 22:38:35.096209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.095780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" event={"ID":"36ba0928-6936-4f74-b6bd-9136788bf5c4","Type":"ContainerDied","Data":"5afe70393c5ba8e1b8accb9d794fda4b80096468483e9f562e17521aac470c55"}
Jul 16 22:38:35.096209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.095942    2581 scope.go:117] "RemoveContainer" containerID="5afe70393c5ba8e1b8accb9d794fda4b80096468483e9f562e17521aac470c55"
Jul 16 22:38:35.097679 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.097658    2581 generic.go:356] "Generic (PLEG): container finished" podID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerID="dc08d4137b2ba0c1df63b67c5cf2058227d884f7038a21b74105c74b048fbc8e" exitCode=6
Jul 16 22:38:35.097792 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.097678    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerDied","Data":"dc08d4137b2ba0c1df63b67c5cf2058227d884f7038a21b74105c74b048fbc8e"}
Jul 16 22:38:35.097792 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.097733    2581 scope.go:117] "RemoveContainer" containerID="7c277d85df72a2d40133d11f09999005d90dc0743c75e9cee18f14a465a17115"
Jul 16 22:38:35.098080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:35.098008    2581 scope.go:117] "RemoveContainer" containerID="dc08d4137b2ba0c1df63b67c5cf2058227d884f7038a21b74105c74b048fbc8e"
Jul 16 22:38:35.098206 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:35.098187    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr_odh-ai-gateway-infra(6feed3ab-e184-40a0-8c64-ba4678d5068f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f"
Jul 16 22:38:36.103845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.103813    2581 generic.go:356] "Generic (PLEG): container finished" podID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerID="d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b" exitCode=6
Jul 16 22:38:36.104274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.103899    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" event={"ID":"36ba0928-6936-4f74-b6bd-9136788bf5c4","Type":"ContainerDied","Data":"d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b"}
Jul 16 22:38:36.104274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.103957    2581 scope.go:117] "RemoveContainer" containerID="5afe70393c5ba8e1b8accb9d794fda4b80096468483e9f562e17521aac470c55"
Jul 16 22:38:36.104274 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.104117    2581 scope.go:117] "RemoveContainer" containerID="d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b"
Jul 16 22:38:36.104457 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:36.104339    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-jqckb_odh-ai-gateway-infra(36ba0928-6936-4f74-b6bd-9136788bf5c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4"
Jul 16 22:38:36.159172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.159144    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:36.281458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.281436    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:36.361878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.361797    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca\") pod \"6feed3ab-e184-40a0-8c64-ba4678d5068f\" (UID: \"6feed3ab-e184-40a0-8c64-ba4678d5068f\") "
Jul 16 22:38:36.362185 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.362157    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6feed3ab-e184-40a0-8c64-ba4678d5068f" (UID: "6feed3ab-e184-40a0-8c64-ba4678d5068f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:36.463190 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:36.463162    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6feed3ab-e184-40a0-8c64-ba4678d5068f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:38:37.111000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.110969    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"
Jul 16 22:38:37.111000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.110985    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr" event={"ID":"6feed3ab-e184-40a0-8c64-ba4678d5068f","Type":"ContainerDied","Data":"71dca83b7a1128013470b6b9866aebf0fc7814411a7e3b0c96c3f468848244e8"}
Jul 16 22:38:37.111534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.111064    2581 scope.go:117] "RemoveContainer" containerID="dc08d4137b2ba0c1df63b67c5cf2058227d884f7038a21b74105c74b048fbc8e"
Jul 16 22:38:37.112974 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.112955    2581 scope.go:117] "RemoveContainer" containerID="d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b"
Jul 16 22:38:37.113171 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:37.113151    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-jqckb_odh-ai-gateway-infra(36ba0928-6936-4f74-b6bd-9136788bf5c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4"
Jul 16 22:38:37.145543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.145517    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:37.147634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.147611    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcffr"]
Jul 16 22:38:37.160418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:37.160394    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" path="/var/lib/kubelet/pods/6feed3ab-e184-40a0-8c64-ba4678d5068f/volumes"
Jul 16 22:38:45.155465 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:45.155432    2581 scope.go:117] "RemoveContainer" containerID="2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2"
Jul 16 22:38:46.162174 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:46.162144    2581 generic.go:356] "Generic (PLEG): container finished" podID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerID="46a431b011877196777737dcfe26194e6e009d3e614ebd2cfbd61a6506da8d7f" exitCode=6
Jul 16 22:38:46.162617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:46.162220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" event={"ID":"adb4aaec-4ee3-45c7-93f3-909f6c528539","Type":"ContainerDied","Data":"46a431b011877196777737dcfe26194e6e009d3e614ebd2cfbd61a6506da8d7f"}
Jul 16 22:38:46.162617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:46.162301    2581 scope.go:117] "RemoveContainer" containerID="2f4bd27347c1e2d2f3f8645887c0385a0f8a70ee1cfa7c5e8aeaa326a1ad83f2"
Jul 16 22:38:46.162617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:46.162585    2581 scope.go:117] "RemoveContainer" containerID="46a431b011877196777737dcfe26194e6e009d3e614ebd2cfbd61a6506da8d7f"
Jul 16 22:38:46.162831 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:46.162809    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2_odh-ai-gateway-infra(adb4aaec-4ee3-45c7-93f3-909f6c528539)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539"
Jul 16 22:38:46.820714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:46.820681    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:38:47.197077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:47.197050    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:47.329402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:47.329380    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:47.452177 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:47.452094    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca\") pod \"adb4aaec-4ee3-45c7-93f3-909f6c528539\" (UID: \"adb4aaec-4ee3-45c7-93f3-909f6c528539\") "
Jul 16 22:38:47.452458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:47.452436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "adb4aaec-4ee3-45c7-93f3-909f6c528539" (UID: "adb4aaec-4ee3-45c7-93f3-909f6c528539"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:47.553765 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:47.553720    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/adb4aaec-4ee3-45c7-93f3-909f6c528539-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:38:48.173120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:48.173093    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"
Jul 16 22:38:48.173326 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:48.173119    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2" event={"ID":"adb4aaec-4ee3-45c7-93f3-909f6c528539","Type":"ContainerDied","Data":"1070d515cccbc816cb36189397cb018250ef57e6112d10b978b2bd730e9e1d39"}
Jul 16 22:38:48.173326 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:48.173171    2581 scope.go:117] "RemoveContainer" containerID="46a431b011877196777737dcfe26194e6e009d3e614ebd2cfbd61a6506da8d7f"
Jul 16 22:38:48.208113 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:48.208086    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:48.214295 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:48.214272    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-l4qn2"]
Jul 16 22:38:49.160454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:49.160421    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" path="/var/lib/kubelet/pods/adb4aaec-4ee3-45c7-93f3-909f6c528539/volumes"
Jul 16 22:38:52.154758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:52.154730    2581 scope.go:117] "RemoveContainer" containerID="d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b"
Jul 16 22:38:53.196924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:53.196841    2581 generic.go:356] "Generic (PLEG): container finished" podID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerID="7c4225833d4b88915c8d1592a77bfb6e670e5086cd72031500f2c64f6c79d1eb" exitCode=6
Jul 16 22:38:53.196924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:53.196891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" event={"ID":"36ba0928-6936-4f74-b6bd-9136788bf5c4","Type":"ContainerDied","Data":"7c4225833d4b88915c8d1592a77bfb6e670e5086cd72031500f2c64f6c79d1eb"}
Jul 16 22:38:53.197470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:53.196938    2581 scope.go:117] "RemoveContainer" containerID="d9751c42cb54ce62a350e6b759cad3fb54ddcae5774544875d4bb8351e84716b"
Jul 16 22:38:53.197470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:53.197304    2581 scope.go:117] "RemoveContainer" containerID="7c4225833d4b88915c8d1592a77bfb6e670e5086cd72031500f2c64f6c79d1eb"
Jul 16 22:38:53.197586 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:53.197540    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-jqckb_odh-ai-gateway-infra(36ba0928-6936-4f74-b6bd-9136788bf5c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4"
Jul 16 22:38:54.232309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:54.232270    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:54.371148 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:54.371122    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:54.514270 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:54.514168    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca\") pod \"36ba0928-6936-4f74-b6bd-9136788bf5c4\" (UID: \"36ba0928-6936-4f74-b6bd-9136788bf5c4\") "
Jul 16 22:38:54.514508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:54.514483    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36ba0928-6936-4f74-b6bd-9136788bf5c4" (UID: "36ba0928-6936-4f74-b6bd-9136788bf5c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:54.614898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:54.614863    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ba0928-6936-4f74-b6bd-9136788bf5c4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:38:55.208438 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.208409    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"
Jul 16 22:38:55.208602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.208415    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb" event={"ID":"36ba0928-6936-4f74-b6bd-9136788bf5c4","Type":"ContainerDied","Data":"d83b7db72a9ec6dbf91d4a5d737af695bf93aa1796d5e5da59840440842a141b"}
Jul 16 22:38:55.208602 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.208539    2581 scope.go:117] "RemoveContainer" containerID="7c4225833d4b88915c8d1592a77bfb6e670e5086cd72031500f2c64f6c79d1eb"
Jul 16 22:38:55.226988 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.226968    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:55.233711 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.233688    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-jqckb"]
Jul 16 22:38:55.818402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818365    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:38:55.818744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818715    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818736    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818745    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.818744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818751    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818759    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818766    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818773    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818780    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818798    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818806    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818815    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818820    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818830    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818835    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818888    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818897    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818902    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818911    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818917    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818922    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="6feed3ab-e184-40a0-8c64-ba4678d5068f" containerName="revoke-keys"
Jul 16 22:38:55.818966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.818927    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:38:55.821421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.821404    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:38:55.823622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.823591    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:38:55.823747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.823702    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:38:55.829001 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.828944    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:38:55.925594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:55.925554    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w\" (UID: \"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:38:56.026482 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:56.026427    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w\" (UID: \"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:38:56.027013 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:56.026995    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w\" (UID: \"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:38:56.133347 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:56.133309    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:38:56.264413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:56.264389    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:38:56.266296 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:38:56.266269    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25ccf9ee_51df_43fe_b7ee_480d93c9a6d4.slice/crio-f2adad4af3aae92a0fd644c60e9a0fc12e1f028800b937c16edbd057ed6cf7db WatchSource:0}: Error finding container f2adad4af3aae92a0fd644c60e9a0fc12e1f028800b937c16edbd057ed6cf7db: Status 404 returned error can't find the container with id f2adad4af3aae92a0fd644c60e9a0fc12e1f028800b937c16edbd057ed6cf7db
Jul 16 22:38:57.159891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:57.159855    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" path="/var/lib/kubelet/pods/36ba0928-6936-4f74-b6bd-9136788bf5c4/volumes"
Jul 16 22:38:57.220183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:57.220100    2581 generic.go:356] "Generic (PLEG): container finished" podID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerID="23d858c204f6f44ed7cfea8d10aed1fe92b0fa580465d9181797a0339e629e16" exitCode=6
Jul 16 22:38:57.220183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:57.220149    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" event={"ID":"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4","Type":"ContainerDied","Data":"23d858c204f6f44ed7cfea8d10aed1fe92b0fa580465d9181797a0339e629e16"}
Jul 16 22:38:57.220390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:57.220204    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" event={"ID":"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4","Type":"ContainerStarted","Data":"f2adad4af3aae92a0fd644c60e9a0fc12e1f028800b937c16edbd057ed6cf7db"}
Jul 16 22:38:57.220461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:57.220445    2581 scope.go:117] "RemoveContainer" containerID="23d858c204f6f44ed7cfea8d10aed1fe92b0fa580465d9181797a0339e629e16"
Jul 16 22:38:58.227432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:58.227351    2581 generic.go:356] "Generic (PLEG): container finished" podID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerID="1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49" exitCode=6
Jul 16 22:38:58.227844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:58.227443    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" event={"ID":"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4","Type":"ContainerDied","Data":"1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49"}
Jul 16 22:38:58.227844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:58.227506    2581 scope.go:117] "RemoveContainer" containerID="23d858c204f6f44ed7cfea8d10aed1fe92b0fa580465d9181797a0339e629e16"
Jul 16 22:38:58.227844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:58.227716    2581 scope.go:117] "RemoveContainer" containerID="1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49"
Jul 16 22:38:58.227981 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:58.227929    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w_odh-ai-gateway-infra(25ccf9ee-51df-43fe-b7ee-480d93c9a6d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4"
Jul 16 22:38:59.233667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:38:59.233639    2581 scope.go:117] "RemoveContainer" containerID="1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49"
Jul 16 22:38:59.234060 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:38:59.233835    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w_odh-ai-gateway-infra(25ccf9ee-51df-43fe-b7ee-480d93c9a6d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4"
Jul 16 22:39:00.724007 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:00.723964    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:39:15.071798 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.071717    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072038    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072049    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072061    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072067    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072127    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="adb4aaec-4ee3-45c7-93f3-909f6c528539" containerName="revoke-keys"
Jul 16 22:39:15.072193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.072134    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ba0928-6936-4f74-b6bd-9136788bf5c4" containerName="revoke-keys"
Jul 16 22:39:15.074908 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.074893    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:15.081913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.081885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:15.155116 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.155087    2581 scope.go:117] "RemoveContainer" containerID="1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49"
Jul 16 22:39:15.199172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.199143    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz\" (UID: \"dce6f9c5-9da9-4a9a-b407-95dd9890799b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:15.299778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.299745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz\" (UID: \"dce6f9c5-9da9-4a9a-b407-95dd9890799b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:15.300414 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.300392    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz\" (UID: \"dce6f9c5-9da9-4a9a-b407-95dd9890799b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:15.386453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.386430    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:15.545849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:15.545827    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:15.548407 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:39:15.548375    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddce6f9c5_9da9_4a9a_b407_95dd9890799b.slice/crio-71b3ef07a0ab2767741778977c070f31dff6cf8ec87586478293e1f8ba1df1cd WatchSource:0}: Error finding container 71b3ef07a0ab2767741778977c070f31dff6cf8ec87586478293e1f8ba1df1cd: Status 404 returned error can't find the container with id 71b3ef07a0ab2767741778977c070f31dff6cf8ec87586478293e1f8ba1df1cd
Jul 16 22:39:16.312499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.312463    2581 generic.go:356] "Generic (PLEG): container finished" podID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerID="33935e1c2ddda88365397d41720e170b64cd658f2d60b32d98354c31736d856f" exitCode=6
Jul 16 22:39:16.312880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.312543    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" event={"ID":"dce6f9c5-9da9-4a9a-b407-95dd9890799b","Type":"ContainerDied","Data":"33935e1c2ddda88365397d41720e170b64cd658f2d60b32d98354c31736d856f"}
Jul 16 22:39:16.312880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.312597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" event={"ID":"dce6f9c5-9da9-4a9a-b407-95dd9890799b","Type":"ContainerStarted","Data":"71b3ef07a0ab2767741778977c070f31dff6cf8ec87586478293e1f8ba1df1cd"}
Jul 16 22:39:16.312880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.312692    2581 scope.go:117] "RemoveContainer" containerID="33935e1c2ddda88365397d41720e170b64cd658f2d60b32d98354c31736d856f"
Jul 16 22:39:16.314502 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.314482    2581 generic.go:356] "Generic (PLEG): container finished" podID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerID="37c94b7ca11b8bc53f195aeaeb609ed98fb5b323372ad8e855ef922428270ba7" exitCode=6
Jul 16 22:39:16.314618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.314516    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" event={"ID":"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4","Type":"ContainerDied","Data":"37c94b7ca11b8bc53f195aeaeb609ed98fb5b323372ad8e855ef922428270ba7"}
Jul 16 22:39:16.314618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.314567    2581 scope.go:117] "RemoveContainer" containerID="1eff0677b9c42acab696b11d2802a476f8287885a0e2ad3479a1827c4bb0fb49"
Jul 16 22:39:16.314813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:16.314799    2581 scope.go:117] "RemoveContainer" containerID="37c94b7ca11b8bc53f195aeaeb609ed98fb5b323372ad8e855ef922428270ba7"
Jul 16 22:39:16.314995 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:16.314972    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w_odh-ai-gateway-infra(25ccf9ee-51df-43fe-b7ee-480d93c9a6d4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4"
Jul 16 22:39:17.322008 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.321919    2581 generic.go:356] "Generic (PLEG): container finished" podID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerID="2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68" exitCode=6
Jul 16 22:39:17.322481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.322006    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" event={"ID":"dce6f9c5-9da9-4a9a-b407-95dd9890799b","Type":"ContainerDied","Data":"2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68"}
Jul 16 22:39:17.322481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.322066    2581 scope.go:117] "RemoveContainer" containerID="33935e1c2ddda88365397d41720e170b64cd658f2d60b32d98354c31736d856f"
Jul 16 22:39:17.322481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.322185    2581 scope.go:117] "RemoveContainer" containerID="2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68"
Jul 16 22:39:17.322481 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:17.322384    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz_odh-ai-gateway-infra(dce6f9c5-9da9-4a9a-b407-95dd9890799b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b"
Jul 16 22:39:17.356200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.356178    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:39:17.487473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.487451    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:39:17.620960 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.620930    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca\") pod \"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4\" (UID: \"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4\") "
Jul 16 22:39:17.621343 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.621314    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" (UID: "25ccf9ee-51df-43fe-b7ee-480d93c9a6d4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:17.721518 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:17.721477    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:18.329680 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.329647    2581 scope.go:117] "RemoveContainer" containerID="2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68"
Jul 16 22:39:18.330086 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:18.329846    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz_odh-ai-gateway-infra(dce6f9c5-9da9-4a9a-b407-95dd9890799b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b"
Jul 16 22:39:18.330739 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.330718    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w" event={"ID":"25ccf9ee-51df-43fe-b7ee-480d93c9a6d4","Type":"ContainerDied","Data":"f2adad4af3aae92a0fd644c60e9a0fc12e1f028800b937c16edbd057ed6cf7db"}
Jul 16 22:39:18.330805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.330770    2581 scope.go:117] "RemoveContainer" containerID="37c94b7ca11b8bc53f195aeaeb609ed98fb5b323372ad8e855ef922428270ba7"
Jul 16 22:39:18.330846 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.330722    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"
Jul 16 22:39:18.367362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.367330    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:39:18.369922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:18.369902    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-2585w"]
Jul 16 22:39:19.160594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:19.160562    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" path="/var/lib/kubelet/pods/25ccf9ee-51df-43fe-b7ee-480d93c9a6d4/volumes"
Jul 16 22:39:27.374558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374520    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374872    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374888    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374893    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374901    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.374948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374907    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.375133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374956    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.375133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.374969    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:27.379170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.379148    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:27.383959 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.383933    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw\" (UID: \"ca254ded-70a5-4542-aee0-4dd82114868d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:27.390537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.390510    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:27.484863 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.484832    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw\" (UID: \"ca254ded-70a5-4542-aee0-4dd82114868d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:27.485445 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.485427    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw\" (UID: \"ca254ded-70a5-4542-aee0-4dd82114868d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:27.690109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.690033    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:27.820919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:27.820888    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:27.823166 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:39:27.823135    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca254ded_70a5_4542_aee0_4dd82114868d.slice/crio-59999fda46f8b1203d10a90bebe10d271a5f297c1a4b56ff81c0c97792e17127 WatchSource:0}: Error finding container 59999fda46f8b1203d10a90bebe10d271a5f297c1a4b56ff81c0c97792e17127: Status 404 returned error can't find the container with id 59999fda46f8b1203d10a90bebe10d271a5f297c1a4b56ff81c0c97792e17127
Jul 16 22:39:28.378725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.378688    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerStarted","Data":"18bade2171579ed6c1484cca4b1f74a388e76757c19ccd735697e3bbcfbf253a"}
Jul 16 22:39:28.379120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.378754    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerStarted","Data":"59999fda46f8b1203d10a90bebe10d271a5f297c1a4b56ff81c0c97792e17127"}
Jul 16 22:39:28.400083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.400030    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" podStartSLOduration=1.400015641 podStartE2EDuration="1.400015641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:28.397804566 +0000 UTC m=+3155.727244469" watchObservedRunningTime="2026-07-16 22:39:28.400015641 +0000 UTC m=+3155.729455545"
Jul 16 22:39:28.609147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.609108    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"]
Jul 16 22:39:28.609579 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.609562    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="25ccf9ee-51df-43fe-b7ee-480d93c9a6d4" containerName="revoke-keys"
Jul 16 22:39:28.612396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.612377    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:28.626128 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.626101    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"]
Jul 16 22:39:28.697807 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.697719    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks\" (UID: \"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:28.798676 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.798640    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks\" (UID: \"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:28.799323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.799304    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks\" (UID: \"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:28.922443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:28.922410    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:29.081358 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.081333    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"]
Jul 16 22:39:29.083184 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:39:29.083149    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda05fa8c5_396e_48a4_9ceb_a77a1b8a3761.slice/crio-2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5 WatchSource:0}: Error finding container 2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5: Status 404 returned error can't find the container with id 2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5
Jul 16 22:39:29.385160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.385127    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks" event={"ID":"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761","Type":"ContainerStarted","Data":"4cab019cb754ba07eca1d3826dc71561e67af2b48a6e1420171ecef5c43e64fd"}
Jul 16 22:39:29.385627 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.385182    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks" event={"ID":"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761","Type":"ContainerStarted","Data":"2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5"}
Jul 16 22:39:29.386653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.386630    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca254ded-70a5-4542-aee0-4dd82114868d" containerID="18bade2171579ed6c1484cca4b1f74a388e76757c19ccd735697e3bbcfbf253a" exitCode=6
Jul 16 22:39:29.386770 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.386720    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerDied","Data":"18bade2171579ed6c1484cca4b1f74a388e76757c19ccd735697e3bbcfbf253a"}
Jul 16 22:39:29.386885 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.386872    2581 scope.go:117] "RemoveContainer" containerID="18bade2171579ed6c1484cca4b1f74a388e76757c19ccd735697e3bbcfbf253a"
Jul 16 22:39:29.402560 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:29.402520    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks" podStartSLOduration=1.402505869 podStartE2EDuration="1.402505869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:29.400913534 +0000 UTC m=+3156.730353438" watchObservedRunningTime="2026-07-16 22:39:29.402505869 +0000 UTC m=+3156.731945771"
Jul 16 22:39:30.154891 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.154815    2581 scope.go:117] "RemoveContainer" containerID="2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68"
Jul 16 22:39:30.393754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.393715    2581 generic.go:356] "Generic (PLEG): container finished" podID="a05fa8c5-396e-48a4-9ceb-a77a1b8a3761" containerID="4cab019cb754ba07eca1d3826dc71561e67af2b48a6e1420171ecef5c43e64fd" exitCode=0
Jul 16 22:39:30.394200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.393775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks" event={"ID":"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761","Type":"ContainerDied","Data":"4cab019cb754ba07eca1d3826dc71561e67af2b48a6e1420171ecef5c43e64fd"}
Jul 16 22:39:30.395543 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.395519    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca254ded-70a5-4542-aee0-4dd82114868d" containerID="739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3" exitCode=6
Jul 16 22:39:30.395646 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.395554    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerDied","Data":"739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3"}
Jul 16 22:39:30.395646 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.395602    2581 scope.go:117] "RemoveContainer" containerID="18bade2171579ed6c1484cca4b1f74a388e76757c19ccd735697e3bbcfbf253a"
Jul 16 22:39:30.395741 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:30.395728    2581 scope.go:117] "RemoveContainer" containerID="739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3"
Jul 16 22:39:30.395955 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:30.395936    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw_odh-ai-gateway-infra(ca254ded-70a5-4542-aee0-4dd82114868d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" podUID="ca254ded-70a5-4542-aee0-4dd82114868d"
Jul 16 22:39:31.401762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.401731    2581 generic.go:356] "Generic (PLEG): container finished" podID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerID="5b13da8370e67f9526b14501be22c8fc95a6ed0487e4a8f5c2495ed8c04ba26a" exitCode=6
Jul 16 22:39:31.402204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.401806    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" event={"ID":"dce6f9c5-9da9-4a9a-b407-95dd9890799b","Type":"ContainerDied","Data":"5b13da8370e67f9526b14501be22c8fc95a6ed0487e4a8f5c2495ed8c04ba26a"}
Jul 16 22:39:31.402204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.401862    2581 scope.go:117] "RemoveContainer" containerID="2e3981a799b58e5a8e0244c8f0b5d7d822f21ee45345e6f0619dd44887602e68"
Jul 16 22:39:31.402204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.402123    2581 scope.go:117] "RemoveContainer" containerID="5b13da8370e67f9526b14501be22c8fc95a6ed0487e4a8f5c2495ed8c04ba26a"
Jul 16 22:39:31.402411 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:31.402367    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz_odh-ai-gateway-infra(dce6f9c5-9da9-4a9a-b407-95dd9890799b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b"
Jul 16 22:39:31.403805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.403787    2581 scope.go:117] "RemoveContainer" containerID="739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3"
Jul 16 22:39:31.404002 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:31.403979    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw_odh-ai-gateway-infra(ca254ded-70a5-4542-aee0-4dd82114868d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" podUID="ca254ded-70a5-4542-aee0-4dd82114868d"
Jul 16 22:39:31.537206 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.537183    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:31.625518 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.625482    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca\") pod \"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761\" (UID: \"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761\") "
Jul 16 22:39:31.625815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.625794    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a05fa8c5-396e-48a4-9ceb-a77a1b8a3761" (UID: "a05fa8c5-396e-48a4-9ceb-a77a1b8a3761"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:31.726141 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:31.726062    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a05fa8c5-396e-48a4-9ceb-a77a1b8a3761-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:32.313783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.313744    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:39:32.410400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.410372    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks"
Jul 16 22:39:32.410759 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.410407    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks" event={"ID":"a05fa8c5-396e-48a4-9ceb-a77a1b8a3761","Type":"ContainerDied","Data":"2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5"}
Jul 16 22:39:32.410759 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.410458    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2d0b83a131887d592eb26369b9f5f867070045563a4b9c7e71ddd1845a566ed5"
Jul 16 22:39:32.427913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.427892    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:32.560458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.560434    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:32.633053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.633025    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca\") pod \"dce6f9c5-9da9-4a9a-b407-95dd9890799b\" (UID: \"dce6f9c5-9da9-4a9a-b407-95dd9890799b\") "
Jul 16 22:39:32.633373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.633353    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dce6f9c5-9da9-4a9a-b407-95dd9890799b" (UID: "dce6f9c5-9da9-4a9a-b407-95dd9890799b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:32.734331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:32.734303    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dce6f9c5-9da9-4a9a-b407-95dd9890799b-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:33.416127 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.416097    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"
Jul 16 22:39:33.416570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.416096    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz" event={"ID":"dce6f9c5-9da9-4a9a-b407-95dd9890799b","Type":"ContainerDied","Data":"71b3ef07a0ab2767741778977c070f31dff6cf8ec87586478293e1f8ba1df1cd"}
Jul 16 22:39:33.416570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.416275    2581 scope.go:117] "RemoveContainer" containerID="5b13da8370e67f9526b14501be22c8fc95a6ed0487e4a8f5c2495ed8c04ba26a"
Jul 16 22:39:33.435556 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.435529    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:33.440242 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.440208    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-7mxfz"]
Jul 16 22:39:33.461077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461050    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:33.461400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461388    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461401    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461413    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461419    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461430    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461435    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a05fa8c5-396e-48a4-9ceb-a77a1b8a3761" containerName="revoke-keys"
Jul 16 22:39:33.461623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461449    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="a05fa8c5-396e-48a4-9ceb-a77a1b8a3761" containerName="revoke-keys"
Jul 16 22:39:33.461623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461503    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="a05fa8c5-396e-48a4-9ceb-a77a1b8a3761" containerName="revoke-keys"
Jul 16 22:39:33.461623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461511    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.461623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.461516    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:33.465664 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.465648    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:33.472904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.472361    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:33.540690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.540659    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-69trh\" (UID: \"8cd14229-44b6-47a8-96af-37e8a5a3e952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:33.641750 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.641708    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-69trh\" (UID: \"8cd14229-44b6-47a8-96af-37e8a5a3e952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:33.642375 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.642357    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-69trh\" (UID: \"8cd14229-44b6-47a8-96af-37e8a5a3e952\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:33.777476 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.777393    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:33.909815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:33.909788    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:33.911468 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:39:33.911436    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cd14229_44b6_47a8_96af_37e8a5a3e952.slice/crio-516a381127272988b631840b85b716a4fd9059d14d30f84166d6eb976f710e35 WatchSource:0}: Error finding container 516a381127272988b631840b85b716a4fd9059d14d30f84166d6eb976f710e35: Status 404 returned error can't find the container with id 516a381127272988b631840b85b716a4fd9059d14d30f84166d6eb976f710e35
Jul 16 22:39:34.424170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:34.424081    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerStarted","Data":"8a1742c63011876a2fdd0fb148360df04b75af756a02eaeaec63bfe3f21ced3e"}
Jul 16 22:39:34.424170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:34.424134    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerStarted","Data":"516a381127272988b631840b85b716a4fd9059d14d30f84166d6eb976f710e35"}
Jul 16 22:39:34.444481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:34.444434    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" podStartSLOduration=1.444420476 podStartE2EDuration="1.444420476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:34.443432956 +0000 UTC m=+3161.772872860" watchObservedRunningTime="2026-07-16 22:39:34.444420476 +0000 UTC m=+3161.773860422"
Jul 16 22:39:35.161325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:35.161293    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" path="/var/lib/kubelet/pods/dce6f9c5-9da9-4a9a-b407-95dd9890799b/volumes"
Jul 16 22:39:35.429963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:35.429879    2581 generic.go:356] "Generic (PLEG): container finished" podID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerID="8a1742c63011876a2fdd0fb148360df04b75af756a02eaeaec63bfe3f21ced3e" exitCode=6
Jul 16 22:39:35.430323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:35.429971    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerDied","Data":"8a1742c63011876a2fdd0fb148360df04b75af756a02eaeaec63bfe3f21ced3e"}
Jul 16 22:39:35.430323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:35.430135    2581 scope.go:117] "RemoveContainer" containerID="8a1742c63011876a2fdd0fb148360df04b75af756a02eaeaec63bfe3f21ced3e"
Jul 16 22:39:36.436102 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:36.436062    2581 generic.go:356] "Generic (PLEG): container finished" podID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerID="5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818" exitCode=6
Jul 16 22:39:36.436540 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:36.436122    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerDied","Data":"5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818"}
Jul 16 22:39:36.436540 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:36.436171    2581 scope.go:117] "RemoveContainer" containerID="8a1742c63011876a2fdd0fb148360df04b75af756a02eaeaec63bfe3f21ced3e"
Jul 16 22:39:36.436540 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:36.436379    2581 scope.go:117] "RemoveContainer" containerID="5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818"
Jul 16 22:39:36.436681 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:36.436610    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-69trh_odh-ai-gateway-infra(8cd14229-44b6-47a8-96af-37e8a5a3e952)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952"
Jul 16 22:39:37.443082 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:37.443056    2581 scope.go:117] "RemoveContainer" containerID="5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818"
Jul 16 22:39:37.443520 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:37.443260    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-69trh_odh-ai-gateway-infra(8cd14229-44b6-47a8-96af-37e8a5a3e952)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952"
Jul 16 22:39:43.157507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:43.157478    2581 scope.go:117] "RemoveContainer" containerID="739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3"
Jul 16 22:39:43.473578 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:43.473474    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerStarted","Data":"86d32a31ef88cf53f06580f31ec04460f0d3ded3b5731015db47c6c1b6d2bc50"}
Jul 16 22:39:44.480568 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:44.480534    2581 generic.go:356] "Generic (PLEG): container finished" podID="ca254ded-70a5-4542-aee0-4dd82114868d" containerID="86d32a31ef88cf53f06580f31ec04460f0d3ded3b5731015db47c6c1b6d2bc50" exitCode=6
Jul 16 22:39:44.481076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:44.480596    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerDied","Data":"86d32a31ef88cf53f06580f31ec04460f0d3ded3b5731015db47c6c1b6d2bc50"}
Jul 16 22:39:44.481076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:44.480656    2581 scope.go:117] "RemoveContainer" containerID="739bdc4ec73f20099718d8b4b0d20a3de17cc38ee3f91922f1fa6f94379aadf3"
Jul 16 22:39:44.481076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:44.480994    2581 scope.go:117] "RemoveContainer" containerID="86d32a31ef88cf53f06580f31ec04460f0d3ded3b5731015db47c6c1b6d2bc50"
Jul 16 22:39:44.481298 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:44.481272    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw_odh-ai-gateway-infra(ca254ded-70a5-4542-aee0-4dd82114868d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" podUID="ca254ded-70a5-4542-aee0-4dd82114868d"
Jul 16 22:39:44.507831 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:44.507796    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:45.619516 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:45.619488    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:45.745590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:45.745488    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca\") pod \"ca254ded-70a5-4542-aee0-4dd82114868d\" (UID: \"ca254ded-70a5-4542-aee0-4dd82114868d\") "
Jul 16 22:39:45.745938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:45.745899    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ca254ded-70a5-4542-aee0-4dd82114868d" (UID: "ca254ded-70a5-4542-aee0-4dd82114868d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:45.847073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:45.847033    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca254ded-70a5-4542-aee0-4dd82114868d-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:46.496378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:46.496341    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw" event={"ID":"ca254ded-70a5-4542-aee0-4dd82114868d","Type":"ContainerDied","Data":"59999fda46f8b1203d10a90bebe10d271a5f297c1a4b56ff81c0c97792e17127"}
Jul 16 22:39:46.496580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:46.496396    2581 scope.go:117] "RemoveContainer" containerID="86d32a31ef88cf53f06580f31ec04460f0d3ded3b5731015db47c6c1b6d2bc50"
Jul 16 22:39:46.496580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:46.496349    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"
Jul 16 22:39:46.524558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:46.524530    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:46.529967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:46.529942    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-mqlrw"]
Jul 16 22:39:47.161161 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:47.161120    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" path="/var/lib/kubelet/pods/ca254ded-70a5-4542-aee0-4dd82114868d/volumes"
Jul 16 22:39:49.155380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:49.155339    2581 scope.go:117] "RemoveContainer" containerID="5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818"
Jul 16 22:39:49.513031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:49.512949    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerStarted","Data":"36a3d3c64ed33b3e1345423c2dc6e6e38f3af2a9cf1282e14ad73c4fd412d7ca"}
Jul 16 22:39:49.924721 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:49.924689    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:39:50.520107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:50.520074    2581 generic.go:356] "Generic (PLEG): container finished" podID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerID="36a3d3c64ed33b3e1345423c2dc6e6e38f3af2a9cf1282e14ad73c4fd412d7ca" exitCode=6
Jul 16 22:39:50.520533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:50.520142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerDied","Data":"36a3d3c64ed33b3e1345423c2dc6e6e38f3af2a9cf1282e14ad73c4fd412d7ca"}
Jul 16 22:39:50.520533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:50.520201    2581 scope.go:117] "RemoveContainer" containerID="5cf8cf39bf6c7c938daf206a1429b61ed05d5017ab46900bb147f3ee8d251818"
Jul 16 22:39:50.520617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:50.520556    2581 scope.go:117] "RemoveContainer" containerID="36a3d3c64ed33b3e1345423c2dc6e6e38f3af2a9cf1282e14ad73c4fd412d7ca"
Jul 16 22:39:50.520732 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:39:50.520710    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-69trh_odh-ai-gateway-infra(8cd14229-44b6-47a8-96af-37e8a5a3e952)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952"
Jul 16 22:39:50.588415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:50.588383    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:51.660096 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:51.660067    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:51.798293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:51.798192    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca\") pod \"8cd14229-44b6-47a8-96af-37e8a5a3e952\" (UID: \"8cd14229-44b6-47a8-96af-37e8a5a3e952\") "
Jul 16 22:39:51.798529 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:51.798506    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8cd14229-44b6-47a8-96af-37e8a5a3e952" (UID: "8cd14229-44b6-47a8-96af-37e8a5a3e952"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:51.899025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:51.898991    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cd14229-44b6-47a8-96af-37e8a5a3e952-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:52.531639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:52.531598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh" event={"ID":"8cd14229-44b6-47a8-96af-37e8a5a3e952","Type":"ContainerDied","Data":"516a381127272988b631840b85b716a4fd9059d14d30f84166d6eb976f710e35"}
Jul 16 22:39:52.531639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:52.531639    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"
Jul 16 22:39:52.531889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:52.531675    2581 scope.go:117] "RemoveContainer" containerID="36a3d3c64ed33b3e1345423c2dc6e6e38f3af2a9cf1282e14ad73c4fd412d7ca"
Jul 16 22:39:52.553079 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:52.553050    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:52.558743 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:52.558721    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-69trh"]
Jul 16 22:39:53.160068 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:53.160034    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" path="/var/lib/kubelet/pods/8cd14229-44b6-47a8-96af-37e8a5a3e952/volumes"
Jul 16 22:39:54.417689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.417658    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:39:54.418136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.417915    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" containerID="cri-o://b47adbfb62bc5f73b3e3b154ff09a8f3d78c11f56aa11aa94b02ced2d8d03d99" gracePeriod=30
Jul 16 22:39:54.785503 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785417    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"]
Jul 16 22:39:54.785826 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785809    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785831    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785852    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785861    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785877    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785887    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785896    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785904    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.785913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785916    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785925    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785933    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.785941    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786027    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786040    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786050    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786061    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cd14229-44b6-47a8-96af-37e8a5a3e952" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786072    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="dce6f9c5-9da9-4a9a-b407-95dd9890799b" containerName="revoke-keys"
Jul 16 22:39:54.786354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.786080    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:39:54.790675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.790656    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:54.793210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.793185    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:39:54.793324 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.793216    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:39:54.797143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.797117    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"]
Jul 16 22:39:54.924506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:54.924475    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8\" (UID: \"9eec6cdb-e319-44db-8e5b-028294782eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:55.025675 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:55.025643    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8\" (UID: \"9eec6cdb-e319-44db-8e5b-028294782eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:55.026312 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:55.026266    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8\" (UID: \"9eec6cdb-e319-44db-8e5b-028294782eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:55.102350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:55.102320    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:55.237774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:55.237750    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"]
Jul 16 22:39:55.240503 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:39:55.240466    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9eec6cdb_e319_44db_8e5b_028294782eee.slice/crio-9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a WatchSource:0}: Error finding container 9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a: Status 404 returned error can't find the container with id 9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a
Jul 16 22:39:55.547714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:55.547679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8" event={"ID":"9eec6cdb-e319-44db-8e5b-028294782eee","Type":"ContainerStarted","Data":"9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a"}
Jul 16 22:39:56.148004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:56.147968    2581 patch_prober.go:28] interesting pod/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:39:56.148180 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:56.148033    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:39:56.559572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:56.559536    2581 generic.go:356] "Generic (PLEG): container finished" podID="9eec6cdb-e319-44db-8e5b-028294782eee" containerID="682447ed416f1dc467c971bfb7921a6e0a5ce6cb8037a47eba9a385a6492a2dd" exitCode=0
Jul 16 22:39:56.559930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:56.559623    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8" event={"ID":"9eec6cdb-e319-44db-8e5b-028294782eee","Type":"ContainerDied","Data":"682447ed416f1dc467c971bfb7921a6e0a5ce6cb8037a47eba9a385a6492a2dd"}
Jul 16 22:39:57.694039 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:57.694015    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:57.848230 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:57.848208    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca\") pod \"9eec6cdb-e319-44db-8e5b-028294782eee\" (UID: \"9eec6cdb-e319-44db-8e5b-028294782eee\") "
Jul 16 22:39:57.848553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:57.848532    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9eec6cdb-e319-44db-8e5b-028294782eee" (UID: "9eec6cdb-e319-44db-8e5b-028294782eee"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:57.948986 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:57.948956    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9eec6cdb-e319-44db-8e5b-028294782eee-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:58.570058 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:58.570021    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8" event={"ID":"9eec6cdb-e319-44db-8e5b-028294782eee","Type":"ContainerDied","Data":"9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a"}
Jul 16 22:39:58.570058 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:58.570056    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8"
Jul 16 22:39:58.570301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:58.570081    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9719109cd3519bd06fed1ef4fd087a34a36de0ff9e6f86fc4e1e61972a3ec26a"
Jul 16 22:39:59.577545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.577515    2581 generic.go:356] "Generic (PLEG): container finished" podID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerID="b47adbfb62bc5f73b3e3b154ff09a8f3d78c11f56aa11aa94b02ced2d8d03d99" exitCode=0
Jul 16 22:39:59.577929 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.577569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerDied","Data":"b47adbfb62bc5f73b3e3b154ff09a8f3d78c11f56aa11aa94b02ced2d8d03d99"}
Jul 16 22:39:59.577929 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.577620    2581 scope.go:117] "RemoveContainer" containerID="3358d700765e754baa7936d842092324784413c99996d989152605bdf760810a"
Jul 16 22:39:59.664681 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.664657    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:39:59.761413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761323    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761360    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761380    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761403    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761431    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8h7c9\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761542    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761578    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761608    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo\") pod \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\" (UID: \"c2737869-15bc-42d7-ab70-e4e3fc455bfe\") "
Jul 16 22:39:59.761753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761668    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:59.762055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761742    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:59.762055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761828    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:59.762055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.761962    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:59.762222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.762060    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.762222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.762082    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.762222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.762097    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.762222 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.762132    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.762644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.762367    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data" (OuterVolumeSpecName: "istio-data") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:59.763775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.763748    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:39:59.763892 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.763874    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:39:59.764374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.764344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token" (OuterVolumeSpecName: "istio-token") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:59.764473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.764344    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9" (OuterVolumeSpecName: "kube-api-access-8h7c9") pod "c2737869-15bc-42d7-ab70-e4e3fc455bfe" (UID: "c2737869-15bc-42d7-ab70-e4e3fc455bfe"). InnerVolumeSpecName "kube-api-access-8h7c9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:39:59.862935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.862909    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.862935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.862932    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.862935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.862940    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.863132 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.862949    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8h7c9\" (UniqueName: \"kubernetes.io/projected/c2737869-15bc-42d7-ab70-e4e3fc455bfe-kube-api-access-8h7c9\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:39:59.863132 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:39:59.862958    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c2737869-15bc-42d7-ab70-e4e3fc455bfe-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:00.582864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:00.582835    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"
Jul 16 22:40:00.583292 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:00.582834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn" event={"ID":"c2737869-15bc-42d7-ab70-e4e3fc455bfe","Type":"ContainerDied","Data":"472784da2ee8280c76a83c05bebd841e8b1189cff409440d7b5c93a8c5a3a568"}
Jul 16 22:40:00.583292 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:00.582964    2581 scope.go:117] "RemoveContainer" containerID="b47adbfb62bc5f73b3e3b154ff09a8f3d78c11f56aa11aa94b02ced2d8d03d99"
Jul 16 22:40:00.608057 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:00.608028    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:40:00.615611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:00.615587    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-3a3e4b-openshift-default-6488d9756f-fjfrn"]
Jul 16 22:40:01.160551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:01.160517    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" path="/var/lib/kubelet/pods/c2737869-15bc-42d7-ab70-e4e3fc455bfe/volumes"
Jul 16 22:40:05.131805 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.131770    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132081    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9eec6cdb-e319-44db-8e5b-028294782eee" containerName="revoke-keys"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132092    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9eec6cdb-e319-44db-8e5b-028294782eee" containerName="revoke-keys"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132103    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132109    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132120    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132126    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132177    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132184    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9eec6cdb-e319-44db-8e5b-028294782eee" containerName="revoke-keys"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132190    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2737869-15bc-42d7-ab70-e4e3fc455bfe" containerName="istio-proxy"
Jul 16 22:40:05.132203 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.132197    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="ca254ded-70a5-4542-aee0-4dd82114868d" containerName="revoke-keys"
Jul 16 22:40:05.134998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.134979    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:05.138138 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.138119    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:40:05.138224 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.138201    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:40:05.147350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.147318    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:05.204608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.204576    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp\" (UID: \"412309ff-e468-4fb7-a365-47de1cc5b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:05.306108 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.306056    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp\" (UID: \"412309ff-e468-4fb7-a365-47de1cc5b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:05.306730 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.306709    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp\" (UID: \"412309ff-e468-4fb7-a365-47de1cc5b9d3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:05.444958 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.444868    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:05.579019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.578980    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:05.582467 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:40:05.582437    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod412309ff_e468_4fb7_a365_47de1cc5b9d3.slice/crio-ff1bb5420dfa4a5f249e6472d9a6c4fcdbd386dec2d5527e212f9736bf659295 WatchSource:0}: Error finding container ff1bb5420dfa4a5f249e6472d9a6c4fcdbd386dec2d5527e212f9736bf659295: Status 404 returned error can't find the container with id ff1bb5420dfa4a5f249e6472d9a6c4fcdbd386dec2d5527e212f9736bf659295
Jul 16 22:40:05.607180 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:05.607142    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerStarted","Data":"ff1bb5420dfa4a5f249e6472d9a6c4fcdbd386dec2d5527e212f9736bf659295"}
Jul 16 22:40:06.612866 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:06.612833    2581 generic.go:356] "Generic (PLEG): container finished" podID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerID="36913c9932316e0e86cf314353822314fc9fe9418c1721ced3ea52b95cdb0269" exitCode=6
Jul 16 22:40:06.613334 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:06.612920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerDied","Data":"36913c9932316e0e86cf314353822314fc9fe9418c1721ced3ea52b95cdb0269"}
Jul 16 22:40:06.613334 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:06.613140    2581 scope.go:117] "RemoveContainer" containerID="36913c9932316e0e86cf314353822314fc9fe9418c1721ced3ea52b95cdb0269"
Jul 16 22:40:07.618910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:07.618873    2581 generic.go:356] "Generic (PLEG): container finished" podID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerID="5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7" exitCode=6
Jul 16 22:40:07.619370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:07.618910    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerDied","Data":"5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7"}
Jul 16 22:40:07.619370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:07.618990    2581 scope.go:117] "RemoveContainer" containerID="36913c9932316e0e86cf314353822314fc9fe9418c1721ced3ea52b95cdb0269"
Jul 16 22:40:07.619370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:07.619150    2581 scope.go:117] "RemoveContainer" containerID="5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7"
Jul 16 22:40:07.619517 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:07.619420    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp_odh-ai-gateway-infra(412309ff-e468-4fb7-a365-47de1cc5b9d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3"
Jul 16 22:40:08.512209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:08.512173    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:40:08.625732 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:08.625700    2581 scope.go:117] "RemoveContainer" containerID="5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7"
Jul 16 22:40:08.626128 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:08.625911    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp_odh-ai-gateway-infra(412309ff-e468-4fb7-a365-47de1cc5b9d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3"
Jul 16 22:40:17.424506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.424470    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:17.428925 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.428909    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:17.438530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.438499    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:17.503551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.503512    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7\" (UID: \"e560d1e0-3dc5-4405-a050-5e31c05adddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:17.604784 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.604748    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7\" (UID: \"e560d1e0-3dc5-4405-a050-5e31c05adddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:17.605420 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.605400    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7\" (UID: \"e560d1e0-3dc5-4405-a050-5e31c05adddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:17.740572 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.740461    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:17.876826 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:17.876794    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:17.880292 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:40:17.880259    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode560d1e0_3dc5_4405_a050_5e31c05adddb.slice/crio-02225067145f03b759fd3a70e5775ee7a3aecfb3a816a9f203d78bba50dda5e1 WatchSource:0}: Error finding container 02225067145f03b759fd3a70e5775ee7a3aecfb3a816a9f203d78bba50dda5e1: Status 404 returned error can't find the container with id 02225067145f03b759fd3a70e5775ee7a3aecfb3a816a9f203d78bba50dda5e1
Jul 16 22:40:18.673313 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:18.673279    2581 generic.go:356] "Generic (PLEG): container finished" podID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerID="033ef00c3f430703ce8e529c0a8df267a1b7bfd30dc52053471fe05714141545" exitCode=6
Jul 16 22:40:18.673731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:18.673362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerDied","Data":"033ef00c3f430703ce8e529c0a8df267a1b7bfd30dc52053471fe05714141545"}
Jul 16 22:40:18.673731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:18.673418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerStarted","Data":"02225067145f03b759fd3a70e5775ee7a3aecfb3a816a9f203d78bba50dda5e1"}
Jul 16 22:40:18.673731 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:18.673550    2581 scope.go:117] "RemoveContainer" containerID="033ef00c3f430703ce8e529c0a8df267a1b7bfd30dc52053471fe05714141545"
Jul 16 22:40:19.679671 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:19.679632    2581 generic.go:356] "Generic (PLEG): container finished" podID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerID="f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3" exitCode=6
Jul 16 22:40:19.680176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:19.679723    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerDied","Data":"f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3"}
Jul 16 22:40:19.680176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:19.679783    2581 scope.go:117] "RemoveContainer" containerID="033ef00c3f430703ce8e529c0a8df267a1b7bfd30dc52053471fe05714141545"
Jul 16 22:40:19.680176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:19.680033    2581 scope.go:117] "RemoveContainer" containerID="f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3"
Jul 16 22:40:19.680401 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:19.680244    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7_odh-ai-gateway-infra(e560d1e0-3dc5-4405-a050-5e31c05adddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb"
Jul 16 22:40:20.554612 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:20.554579    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:40:20.554860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:20.554834    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy" containerID="cri-o://bedbde96d81c3773a4787f0f469fb1707c4da462f2cf197a2a533b2107c3a825" gracePeriod=30
Jul 16 22:40:20.686966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:20.686939    2581 scope.go:117] "RemoveContainer" containerID="f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3"
Jul 16 22:40:20.687403 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:20.687107    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7_odh-ai-gateway-infra(e560d1e0-3dc5-4405-a050-5e31c05adddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb"
Jul 16 22:40:21.478027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.477987    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:40:21.481785 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.481766    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.484319 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.484270    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-a0061f-openshift-default-dockercfg-72gm7\""
Jul 16 22:40:21.494083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.494059    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:40:21.636542 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636502    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636542 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636539    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqb2m\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636577    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636665    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636706    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636736    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.636809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636792    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.637081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636828    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.637081 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.636848    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738042    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738085    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738110    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738143    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738162    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738179    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738209    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738255    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqb2m\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738684    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.738757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738728    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.739071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738830    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.739071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738961    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.739071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.738986    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.741160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.741134    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.741305 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.741264    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.748469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.748435    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqb2m\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.748605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.748521    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token\") pod \"e2e-api-a-a0061f-openshift-default-86d595549f-2p4js\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") " pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.793004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.792963    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:21.934121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.934088    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:40:21.937595 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:40:21.937549    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd799c5a7_bb6b_491e_916c_f19d7512841c.slice/crio-b364b814db14b1b8c36bdc2d9b0f58567143505705de84fc28155c7d6c7ab8c4 WatchSource:0}: Error finding container b364b814db14b1b8c36bdc2d9b0f58567143505705de84fc28155c7d6c7ab8c4: Status 404 returned error can't find the container with id b364b814db14b1b8c36bdc2d9b0f58567143505705de84fc28155c7d6c7ab8c4
Jul 16 22:40:21.939912 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.939860    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:21.940037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.939945    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:21.940037 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:21.939988    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:22.698205 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:22.698172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerStarted","Data":"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"}
Jul 16 22:40:22.698435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:22.698260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerStarted","Data":"b364b814db14b1b8c36bdc2d9b0f58567143505705de84fc28155c7d6c7ab8c4"}
Jul 16 22:40:22.721597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:22.721543    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podStartSLOduration=1.721529768 podStartE2EDuration="1.721529768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:22.720934931 +0000 UTC m=+3210.050374833" watchObservedRunningTime="2026-07-16 22:40:22.721529768 +0000 UTC m=+3210.050969670"
Jul 16 22:40:22.794034 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:22.794003    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:22.798922 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:22.798899    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:23.156662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:23.156631    2581 scope.go:117] "RemoveContainer" containerID="5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7"
Jul 16 22:40:23.705178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:23.705145    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerStarted","Data":"af15b0744c7d66b5f848cb55ff90fe3586644bae820ecabc8f7e7d5f32f97bf5"}
Jul 16 22:40:23.705546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:23.705519    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:23.706718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:23.706697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:23.721781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:23.721730    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" podStartSLOduration=18.721713126 podStartE2EDuration="18.721713126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:23.72134521 +0000 UTC m=+3211.050785116" watchObservedRunningTime="2026-07-16 22:40:23.721713126 +0000 UTC m=+3211.051153032"
Jul 16 22:40:24.711282 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:24.711252    2581 generic.go:356] "Generic (PLEG): container finished" podID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerID="af15b0744c7d66b5f848cb55ff90fe3586644bae820ecabc8f7e7d5f32f97bf5" exitCode=6
Jul 16 22:40:24.711763 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:24.711317    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerDied","Data":"af15b0744c7d66b5f848cb55ff90fe3586644bae820ecabc8f7e7d5f32f97bf5"}
Jul 16 22:40:24.711763 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:24.711398    2581 scope.go:117] "RemoveContainer" containerID="5476da4a92b45c60fa7c9b242a98c277312e6ff0254277179cf402f7db9c65e7"
Jul 16 22:40:24.711763 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:24.711736    2581 scope.go:117] "RemoveContainer" containerID="af15b0744c7d66b5f848cb55ff90fe3586644bae820ecabc8f7e7d5f32f97bf5"
Jul 16 22:40:24.712050 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:24.712019    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp_odh-ai-gateway-infra(412309ff-e468-4fb7-a365-47de1cc5b9d3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3"
Jul 16 22:40:24.739598 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:24.739574    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:25.720677 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.720645    2581 generic.go:356] "Generic (PLEG): container finished" podID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerID="bedbde96d81c3773a4787f0f469fb1707c4da462f2cf197a2a533b2107c3a825" exitCode=0
Jul 16 22:40:25.721171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.720713    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerDied","Data":"bedbde96d81c3773a4787f0f469fb1707c4da462f2cf197a2a533b2107c3a825"}
Jul 16 22:40:25.721171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.720777    2581 scope.go:117] "RemoveContainer" containerID="94649719379d1d7c41468cf82b8d79c4cb7589c5935b59d3cb64cb531b2be882"
Jul 16 22:40:25.820410 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.820341    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:40:25.866316 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.866286    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:25.978480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978443    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978534    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4cwh5\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978555    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978579    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978620    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978635    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.978685 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978678    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.979011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978711    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.979011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978733    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy\") pod \"c1919c46-f394-4c22-8a9e-1841ea53e13c\" (UID: \"c1919c46-f394-4c22-8a9e-1841ea53e13c\") "
Jul 16 22:40:25.979011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978759    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca\") pod \"412309ff-e468-4fb7-a365-47de1cc5b9d3\" (UID: \"412309ff-e468-4fb7-a365-47de1cc5b9d3\") "
Jul 16 22:40:25.979011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978969    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:25.979011 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.978988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:25.979293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.979028    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:25.979293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.979269    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:25.979421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.979338    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "412309ff-e468-4fb7-a365-47de1cc5b9d3" (UID: "412309ff-e468-4fb7-a365-47de1cc5b9d3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:25.979624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.979597    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data" (OuterVolumeSpecName: "istio-data") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:25.981130 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.981102    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token" (OuterVolumeSpecName: "istio-token") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:25.981256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.981139    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:40:25.981256 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.981188    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:40:25.981428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:25.981411    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5" (OuterVolumeSpecName: "kube-api-access-4cwh5") pod "c1919c46-f394-4c22-8a9e-1841ea53e13c" (UID: "c1919c46-f394-4c22-8a9e-1841ea53e13c"). InnerVolumeSpecName "kube-api-access-4cwh5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080126    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080155    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cwh5\" (UniqueName: \"kubernetes.io/projected/c1919c46-f394-4c22-8a9e-1841ea53e13c-kube-api-access-4cwh5\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080165    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080174    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080183    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1919c46-f394-4c22-8a9e-1841ea53e13c-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080191    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080199    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080208    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080216    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1919c46-f394-4c22-8a9e-1841ea53e13c-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.080633 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.080224    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/412309ff-e468-4fb7-a365-47de1cc5b9d3-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:26.726678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.726639    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp" event={"ID":"412309ff-e468-4fb7-a365-47de1cc5b9d3","Type":"ContainerDied","Data":"ff1bb5420dfa4a5f249e6472d9a6c4fcdbd386dec2d5527e212f9736bf659295"}
Jul 16 22:40:26.727172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.726706    2581 scope.go:117] "RemoveContainer" containerID="af15b0744c7d66b5f848cb55ff90fe3586644bae820ecabc8f7e7d5f32f97bf5"
Jul 16 22:40:26.727172 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.726646    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"
Jul 16 22:40:26.728127 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.728106    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"
Jul 16 22:40:26.728208 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.728101    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz" event={"ID":"c1919c46-f394-4c22-8a9e-1841ea53e13c","Type":"ContainerDied","Data":"136352ef42e01e5a1cb91d4355ff95e411166cb3c21b1b2102bcf090c71065b8"}
Jul 16 22:40:26.735561 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.735539    2581 scope.go:117] "RemoveContainer" containerID="bedbde96d81c3773a4787f0f469fb1707c4da462f2cf197a2a533b2107c3a825"
Jul 16 22:40:26.772323 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.772294    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:26.792504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.792482    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-22qlp"]
Jul 16 22:40:26.809223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.809190    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:40:26.815199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:26.815177    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2fe960-openshift-default-89cb555-k7xcz"]
Jul 16 22:40:27.160402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:27.160369    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" path="/var/lib/kubelet/pods/412309ff-e468-4fb7-a365-47de1cc5b9d3/volumes"
Jul 16 22:40:27.160815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:27.160802    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" path="/var/lib/kubelet/pods/c1919c46-f394-4c22-8a9e-1841ea53e13c/volumes"
Jul 16 22:40:29.007670 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.007635    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:40:29.908918 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.908876    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:40:29.909318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909300    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:29.909318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909319    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:29.909474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909337    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909345    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909373    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909382    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909486    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909501    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909510    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:29.909666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.909523    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:29.914530 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.914508    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:29.918462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.918442    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-24b876-openshift-default-dockercfg-fxvlf\""
Jul 16 22:40:29.931867 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:29.931763    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:40:30.015573 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015627    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015663    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015690    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25s9h\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015721    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015754    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015793    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015857    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.016048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.015932    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.116958 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.116916    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.116968    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117004    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117037    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117191    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117307    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117356    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117385    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117615 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117413    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25s9h\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117615 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117461    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117615 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117533    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117774 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117704    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.117901 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.117884    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.119860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.119836    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.120065 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.120034    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.126909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.126881    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.127287 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.127217    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-25s9h\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h\") pod \"e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") " pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.225097 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.225004    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:30.603051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.603016    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:40:30.606565 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:40:30.606538    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod686a8934_16c8_4e3d_b05c_f2e8fe6c0486.slice/crio-5cace5ddd092811b5675bc07a8d6e4b56aba63895918517c6a7d74bce5c0d71a WatchSource:0}: Error finding container 5cace5ddd092811b5675bc07a8d6e4b56aba63895918517c6a7d74bce5c0d71a: Status 404 returned error can't find the container with id 5cace5ddd092811b5675bc07a8d6e4b56aba63895918517c6a7d74bce5c0d71a
Jul 16 22:40:30.608692 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.608662    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:30.608790 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.608725    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:30.608790 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.608755    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:30.753262 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.753195    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerStarted","Data":"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"}
Jul 16 22:40:30.753479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:30.753293    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerStarted","Data":"5cace5ddd092811b5675bc07a8d6e4b56aba63895918517c6a7d74bce5c0d71a"}
Jul 16 22:40:31.225863 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:31.225817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:31.227322 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:31.227294    2581 patch_prober.go:28] interesting pod/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 16 22:40:31.227460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:31.227367    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 16 22:40:32.155819 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.155786    2581 scope.go:117] "RemoveContainer" containerID="f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3"
Jul 16 22:40:32.230307 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.230277    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:32.261084 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.260999    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podStartSLOduration=3.260979736 podStartE2EDuration="3.260979736s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:30.776600856 +0000 UTC m=+3218.106040761" watchObservedRunningTime="2026-07-16 22:40:32.260979736 +0000 UTC m=+3219.590419641"
Jul 16 22:40:32.776337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.776252    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerStarted","Data":"a78fd8e60fab12318f24a4c68ff3839975c98866ff33fa065942c6d80e510ff2"}
Jul 16 22:40:32.776661 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.776626    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:32.777390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.777373    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:32.794882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:32.794832    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" podStartSLOduration=15.794819681 podStartE2EDuration="15.794819681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:32.792454341 +0000 UTC m=+3220.121894255" watchObservedRunningTime="2026-07-16 22:40:32.794819681 +0000 UTC m=+3220.124259606"
Jul 16 22:40:33.782318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:33.782286    2581 generic.go:356] "Generic (PLEG): container finished" podID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerID="a78fd8e60fab12318f24a4c68ff3839975c98866ff33fa065942c6d80e510ff2" exitCode=6
Jul 16 22:40:33.782757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:33.782366    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerDied","Data":"a78fd8e60fab12318f24a4c68ff3839975c98866ff33fa065942c6d80e510ff2"}
Jul 16 22:40:33.782757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:33.782429    2581 scope.go:117] "RemoveContainer" containerID="f9f931875f55b3658d7a85d8d3dd3e9249e5e3836a429d6e10cd51d58820dfc3"
Jul 16 22:40:33.782887 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:33.782828    2581 scope.go:117] "RemoveContainer" containerID="a78fd8e60fab12318f24a4c68ff3839975c98866ff33fa065942c6d80e510ff2"
Jul 16 22:40:33.783051 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:33.783025    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7_odh-ai-gateway-infra(e560d1e0-3dc5-4405-a050-5e31c05adddb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb"
Jul 16 22:40:33.808065 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:33.808042    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:34.930806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:34.930777    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:34.971372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:34.971324    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca\") pod \"e560d1e0-3dc5-4405-a050-5e31c05adddb\" (UID: \"e560d1e0-3dc5-4405-a050-5e31c05adddb\") "
Jul 16 22:40:34.971662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:34.971635    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e560d1e0-3dc5-4405-a050-5e31c05adddb" (UID: "e560d1e0-3dc5-4405-a050-5e31c05adddb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:35.072874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.072782    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e560d1e0-3dc5-4405-a050-5e31c05adddb-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:40:35.794811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.794781    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"
Jul 16 22:40:35.794991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.794780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7" event={"ID":"e560d1e0-3dc5-4405-a050-5e31c05adddb","Type":"ContainerDied","Data":"02225067145f03b759fd3a70e5775ee7a3aecfb3a816a9f203d78bba50dda5e1"}
Jul 16 22:40:35.794991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.794913    2581 scope.go:117] "RemoveContainer" containerID="a78fd8e60fab12318f24a4c68ff3839975c98866ff33fa065942c6d80e510ff2"
Jul 16 22:40:35.812474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.812449    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:35.819424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:35.819400    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-6p7k7"]
Jul 16 22:40:37.160489 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.160415    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" path="/var/lib/kubelet/pods/e560d1e0-3dc5-4405-a050-5e31c05adddb/volumes"
Jul 16 22:40:37.807468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.807431    2581 generic.go:356] "Generic (PLEG): container finished" podID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerID="69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec" exitCode=0
Jul 16 22:40:37.807653 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.807504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerDied","Data":"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"}
Jul 16 22:40:37.807877 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.807863    2581 scope.go:117] "RemoveContainer" containerID="69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"
Jul 16 22:40:37.808808 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.808778    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:37.808864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.808845    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:37.808903 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:37.808875    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:38.705380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:38.705348    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:38.706880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:38.706860    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:40:38.813971 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:38.813941    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerStarted","Data":"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"}
Jul 16 22:40:39.793781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:39.793733    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:39.794909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:39.794875    2581 patch_prober.go:28] interesting pod/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 16 22:40:39.795069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:39.794949    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 16 22:40:40.794452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:40.793734    2581 patch_prober.go:28] interesting pod/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 16 22:40:40.794452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:40.793797    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 16 22:40:41.793649 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:41.793610    2581 patch_prober.go:28] interesting pod/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 16 22:40:41.793954 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:41.793923    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 16 22:40:42.794202 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:42.794156    2581 patch_prober.go:28] interesting pod/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused" start-of-body=
Jul 16 22:40:42.794636 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:42.794245    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.118:15021/healthz/ready\": dial tcp 10.133.0.118:15021: connect: connection refused"
Jul 16 22:40:43.797717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:43.797679    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:43.798175 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:43.797960    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:43.798724 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:43.798709    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:40:45.916469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.916375    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:40:45.917860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.917834    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918020 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918008    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918104    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:45.918185 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918177    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="412309ff-e468-4fb7-a365-47de1cc5b9d3" containerName="revoke-keys"
Jul 16 22:40:45.918310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918299    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918392 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918381    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918468    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:45.918547 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918539    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:45.918622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918614    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918684 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918677    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.918902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918890    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c1919c46-f394-4c22-8a9e-1841ea53e13c" containerName="istio-proxy"
Jul 16 22:40:45.918991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.918982    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.919068 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.919060    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:40:45.925855 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.925816    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:40:45.928797 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.928775    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:40:45.928797 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.928791    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:40:45.940850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.940821    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:40:45.973359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:45.973316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756\" (UID: \"4d1c8689-3138-40d5-b9c5-a104298e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:40:46.073924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.073889    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756\" (UID: \"4d1c8689-3138-40d5-b9c5-a104298e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:40:46.074761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.074737    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756\" (UID: \"4d1c8689-3138-40d5-b9c5-a104298e6310\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:40:46.241359 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.241274    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:40:46.379132 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.379093    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:40:46.382462 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:40:46.382428    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1c8689_3138_40d5_b9c5_a104298e6310.slice/crio-06292df0594b36852d7af71ea8e79fe5f3d4285275ca3f2c235c1111067c438c WatchSource:0}: Error finding container 06292df0594b36852d7af71ea8e79fe5f3d4285275ca3f2c235c1111067c438c: Status 404 returned error can't find the container with id 06292df0594b36852d7af71ea8e79fe5f3d4285275ca3f2c235c1111067c438c
Jul 16 22:40:46.855246 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.855183    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerStarted","Data":"22c8e3f9e9e278416ff5699879f9a392ef65cff59805fdb48b7585b3955b181a"}
Jul 16 22:40:46.855498 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.855278    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerStarted","Data":"06292df0594b36852d7af71ea8e79fe5f3d4285275ca3f2c235c1111067c438c"}
Jul 16 22:40:46.857168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.857140    2581 generic.go:356] "Generic (PLEG): container finished" podID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerID="f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085" exitCode=0
Jul 16 22:40:46.857324 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.857206    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerDied","Data":"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"}
Jul 16 22:40:46.857553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.857540    2581 scope.go:117] "RemoveContainer" containerID="f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"
Jul 16 22:40:46.858584 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.858554    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:46.858694 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.858618    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:46.858694 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.858649    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:40:46.872074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:46.872034    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" podStartSLOduration=1.872019984 podStartE2EDuration="1.872019984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:46.870067443 +0000 UTC m=+3234.199507370" watchObservedRunningTime="2026-07-16 22:40:46.872019984 +0000 UTC m=+3234.201459886"
Jul 16 22:40:47.776754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:47.776718    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:47.867018 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:47.866974    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerID="22c8e3f9e9e278416ff5699879f9a392ef65cff59805fdb48b7585b3955b181a" exitCode=6
Jul 16 22:40:47.867248 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:47.867079    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerDied","Data":"22c8e3f9e9e278416ff5699879f9a392ef65cff59805fdb48b7585b3955b181a"}
Jul 16 22:40:47.867519 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:47.867493    2581 scope.go:117] "RemoveContainer" containerID="22c8e3f9e9e278416ff5699879f9a392ef65cff59805fdb48b7585b3955b181a"
Jul 16 22:40:47.870471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:47.870441    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerStarted","Data":"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"}
Jul 16 22:40:48.225846 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.225812    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:48.227079 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.227047    2581 patch_prober.go:28] interesting pod/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 16 22:40:48.227218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.227097    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 16 22:40:48.877197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.877161    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerID="122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7" exitCode=6
Jul 16 22:40:48.877745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.877260    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerDied","Data":"122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7"}
Jul 16 22:40:48.877745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.877322    2581 scope.go:117] "RemoveContainer" containerID="22c8e3f9e9e278416ff5699879f9a392ef65cff59805fdb48b7585b3955b181a"
Jul 16 22:40:48.877745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:48.877561    2581 scope.go:117] "RemoveContainer" containerID="122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7"
Jul 16 22:40:48.877914 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:48.877761    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756_odh-ai-gateway-infra(4d1c8689-3138-40d5-b9c5-a104298e6310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310"
Jul 16 22:40:49.226213 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:49.226181    2581 patch_prober.go:28] interesting pod/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 16 22:40:49.226409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:49.226254    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 16 22:40:49.885849 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:49.885815    2581 scope.go:117] "RemoveContainer" containerID="122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7"
Jul 16 22:40:49.886273 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:40:49.886047    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756_odh-ai-gateway-infra(4d1c8689-3138-40d5-b9c5-a104298e6310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310"
Jul 16 22:40:50.225728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:50.225639    2581 patch_prober.go:28] interesting pod/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 16 22:40:50.225728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:50.225696    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 16 22:40:51.225616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:51.225576    2581 patch_prober.go:28] interesting pod/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body=
Jul 16 22:40:51.226099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:51.225639    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused"
Jul 16 22:40:52.229363 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:52.229330    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:52.229824 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:52.229618    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:52.230276 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:52.230259    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:40:59.310649 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:40:59.310614    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:41:03.156811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:03.156782    2581 scope.go:117] "RemoveContainer" containerID="122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7"
Jul 16 22:41:03.956400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:03.956319    2581 generic.go:356] "Generic (PLEG): container finished" podID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerID="c83a2c35a4f64b88cb92a027f962d0b351c51098065dd9b54f4bf2634555de87" exitCode=6
Jul 16 22:41:03.956534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:03.956395    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerDied","Data":"c83a2c35a4f64b88cb92a027f962d0b351c51098065dd9b54f4bf2634555de87"}
Jul 16 22:41:03.956534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:03.956459    2581 scope.go:117] "RemoveContainer" containerID="122f149f68352108c1f605f2d0dd1038aab5cfa078b491b2550c3a6ea3c5ebb7"
Jul 16 22:41:03.956755 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:03.956739    2581 scope.go:117] "RemoveContainer" containerID="c83a2c35a4f64b88cb92a027f962d0b351c51098065dd9b54f4bf2634555de87"
Jul 16 22:41:03.956972 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:03.956949    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756_odh-ai-gateway-infra(4d1c8689-3138-40d5-b9c5-a104298e6310)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310"
Jul 16 22:41:04.982487 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:04.982452    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:41:05.109154 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.109132    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:41:05.244252 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.244159    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca\") pod \"4d1c8689-3138-40d5-b9c5-a104298e6310\" (UID: \"4d1c8689-3138-40d5-b9c5-a104298e6310\") "
Jul 16 22:41:05.244557 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.244538    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d1c8689-3138-40d5-b9c5-a104298e6310" (UID: "4d1c8689-3138-40d5-b9c5-a104298e6310"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:05.345532 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.345503    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d1c8689-3138-40d5-b9c5-a104298e6310-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:05.968651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.968619    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"
Jul 16 22:41:05.968865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.968666    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756" event={"ID":"4d1c8689-3138-40d5-b9c5-a104298e6310","Type":"ContainerDied","Data":"06292df0594b36852d7af71ea8e79fe5f3d4285275ca3f2c235c1111067c438c"}
Jul 16 22:41:05.968865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.968734    2581 scope.go:117] "RemoveContainer" containerID="c83a2c35a4f64b88cb92a027f962d0b351c51098065dd9b54f4bf2634555de87"
Jul 16 22:41:05.988440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.988414    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:41:05.997508 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:05.997479    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-lg756"]
Jul 16 22:41:07.159955 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.159909    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" path="/var/lib/kubelet/pods/4d1c8689-3138-40d5-b9c5-a104298e6310/volumes"
Jul 16 22:41:07.475564 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475486    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:07.475814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475803    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.475861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475816    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.475861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475831    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.475861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475837    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.475861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475846    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.475861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475852    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.476012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475901    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.476012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475909    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:07.476012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.475916    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e560d1e0-3dc5-4405-a050-5e31c05adddb" containerName="revoke-keys"
Jul 16 22:41:07.478876 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.478861    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:07.482293 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.482277    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:41:07.482497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.482478    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:41:07.488484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.488461    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:07.561676 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.561643    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw\" (UID: \"264aa26d-83ea-418b-9df3-ed76291777c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:07.662881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.662851    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw\" (UID: \"264aa26d-83ea-418b-9df3-ed76291777c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:07.663485 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.663467    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw\" (UID: \"264aa26d-83ea-418b-9df3-ed76291777c0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:07.789910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.789839    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:07.919593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.919566    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:07.921401 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:07.921369    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod264aa26d_83ea_418b_9df3_ed76291777c0.slice/crio-1f59fbca8a68ff1c45c576be9b891e2ae63c05755247c411bb6c12c97a8d0974 WatchSource:0}: Error finding container 1f59fbca8a68ff1c45c576be9b891e2ae63c05755247c411bb6c12c97a8d0974: Status 404 returned error can't find the container with id 1f59fbca8a68ff1c45c576be9b891e2ae63c05755247c411bb6c12c97a8d0974
Jul 16 22:41:07.983269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:07.983207    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" event={"ID":"264aa26d-83ea-418b-9df3-ed76291777c0","Type":"ContainerStarted","Data":"1f59fbca8a68ff1c45c576be9b891e2ae63c05755247c411bb6c12c97a8d0974"}
Jul 16 22:41:08.990330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:08.990296    2581 generic.go:356] "Generic (PLEG): container finished" podID="264aa26d-83ea-418b-9df3-ed76291777c0" containerID="d0e7360a314e741e0571c9fd365fde385c3996d63ede75004177beb19b823a6f" exitCode=6
Jul 16 22:41:08.990715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:08.990371    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" event={"ID":"264aa26d-83ea-418b-9df3-ed76291777c0","Type":"ContainerDied","Data":"d0e7360a314e741e0571c9fd365fde385c3996d63ede75004177beb19b823a6f"}
Jul 16 22:41:08.990715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:08.990526    2581 scope.go:117] "RemoveContainer" containerID="d0e7360a314e741e0571c9fd365fde385c3996d63ede75004177beb19b823a6f"
Jul 16 22:41:09.996998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:09.996958    2581 generic.go:356] "Generic (PLEG): container finished" podID="264aa26d-83ea-418b-9df3-ed76291777c0" containerID="3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a" exitCode=6
Jul 16 22:41:09.997428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:09.997040    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" event={"ID":"264aa26d-83ea-418b-9df3-ed76291777c0","Type":"ContainerDied","Data":"3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a"}
Jul 16 22:41:09.997428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:09.997132    2581 scope.go:117] "RemoveContainer" containerID="d0e7360a314e741e0571c9fd365fde385c3996d63ede75004177beb19b823a6f"
Jul 16 22:41:09.997428 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:09.997192    2581 scope.go:117] "RemoveContainer" containerID="3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a"
Jul 16 22:41:09.997551 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:09.997436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw_odh-ai-gateway-infra(264aa26d-83ea-418b-9df3-ed76291777c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" podUID="264aa26d-83ea-418b-9df3-ed76291777c0"
Jul 16 22:41:11.004057 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.004031    2581 scope.go:117] "RemoveContainer" containerID="3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a"
Jul 16 22:41:11.004924 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:11.004456    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw_odh-ai-gateway-infra(264aa26d-83ea-418b-9df3-ed76291777c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" podUID="264aa26d-83ea-418b-9df3-ed76291777c0"
Jul 16 22:41:11.006608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.006584    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:41:11.006787 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.006744    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" containerID="cri-o://cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87" gracePeriod=30
Jul 16 22:41:11.372271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.372241    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"]
Jul 16 22:41:11.372666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.372652    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d1c8689-3138-40d5-b9c5-a104298e6310" containerName="revoke-keys"
Jul 16 22:41:11.376493 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.376477    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:11.385178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.385153    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"]
Jul 16 22:41:11.494148 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.494114    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24b876-5jvd7\" (UID: \"928cf688-cc12-4150-8285-85f2ee813e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:11.594822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.594794    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24b876-5jvd7\" (UID: \"928cf688-cc12-4150-8285-85f2ee813e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:11.595370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.595345    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-24b876-5jvd7\" (UID: \"928cf688-cc12-4150-8285-85f2ee813e41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:11.687290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.687192    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:11.817781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:11.817686    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"]
Jul 16 22:41:11.819906 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:11.819877    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod928cf688_cc12_4150_8285_85f2ee813e41.slice/crio-d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c WatchSource:0}: Error finding container d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c: Status 404 returned error can't find the container with id d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c
Jul 16 22:41:12.009634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:12.009606    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7" event={"ID":"928cf688-cc12-4150-8285-85f2ee813e41","Type":"ContainerStarted","Data":"d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c"}
Jul 16 22:41:13.016004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:13.015916    2581 generic.go:356] "Generic (PLEG): container finished" podID="928cf688-cc12-4150-8285-85f2ee813e41" containerID="64d74d3c000caaa99edca153c24e95dcbfa5712bb3697c69fdf0fd52b5bafe9d" exitCode=0
Jul 16 22:41:13.016004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:13.015986    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7" event={"ID":"928cf688-cc12-4150-8285-85f2ee813e41","Type":"ContainerDied","Data":"64d74d3c000caaa99edca153c24e95dcbfa5712bb3697c69fdf0fd52b5bafe9d"}
Jul 16 22:41:13.799952 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:13.799919    2581 patch_prober.go:28] interesting pod/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:41:13.800121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:13.799975    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:41:14.166387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:14.166364    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:14.316559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:14.316523    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca\") pod \"928cf688-cc12-4150-8285-85f2ee813e41\" (UID: \"928cf688-cc12-4150-8285-85f2ee813e41\") "
Jul 16 22:41:14.316864 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:14.316844    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "928cf688-cc12-4150-8285-85f2ee813e41" (UID: "928cf688-cc12-4150-8285-85f2ee813e41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:14.417149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:14.417121    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/928cf688-cc12-4150-8285-85f2ee813e41-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:15.026606 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:15.026578    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7"
Jul 16 22:41:15.026781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:15.026571    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-24b876-5jvd7" event={"ID":"928cf688-cc12-4150-8285-85f2ee813e41","Type":"ContainerDied","Data":"d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c"}
Jul 16 22:41:15.026781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:15.026736    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4e5fb561a59a0c5b7e591ca04a748179b52cd41912ffaa6eb84dd6ecc9a1c6c"
Jul 16 22:41:16.256592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.256569    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:41:16.432533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432502    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqb2m\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432561    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432707 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432700    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432749    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432771    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432798    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432803    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432822    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.432889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.432849    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy\") pod \"d799c5a7-bb6b-491e-916c-f19d7512841c\" (UID: \"d799c5a7-bb6b-491e-916c-f19d7512841c\") "
Jul 16 22:41:16.433188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.433008    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:16.433188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.433073    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:16.433188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.433142    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:16.433188 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.433159    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.433415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.433176    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.434332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.434305    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data" (OuterVolumeSpecName: "istio-data") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:16.434920 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.434892    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m" (OuterVolumeSpecName: "kube-api-access-rqb2m") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "kube-api-access-rqb2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:16.435054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.434977    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:41:16.435443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.435425    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token" (OuterVolumeSpecName: "istio-token") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:16.435520 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.435424    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d799c5a7-bb6b-491e-916c-f19d7512841c" (UID: "d799c5a7-bb6b-491e-916c-f19d7512841c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:16.533996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.533971    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d799c5a7-bb6b-491e-916c-f19d7512841c-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.533996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.533993    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.534144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.534002    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.534144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.534011    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.534144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.534019    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.534144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.534027    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqb2m\" (UniqueName: \"kubernetes.io/projected/d799c5a7-bb6b-491e-916c-f19d7512841c-kube-api-access-rqb2m\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:16.534144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:16.534035    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d799c5a7-bb6b-491e-916c-f19d7512841c-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:17.036571 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.036535    2581 generic.go:356] "Generic (PLEG): container finished" podID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerID="cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87" exitCode=0
Jul 16 22:41:17.036762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.036583    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"
Jul 16 22:41:17.036762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.036619    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerDied","Data":"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"}
Jul 16 22:41:17.036762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.036673    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js" event={"ID":"d799c5a7-bb6b-491e-916c-f19d7512841c","Type":"ContainerDied","Data":"b364b814db14b1b8c36bdc2d9b0f58567143505705de84fc28155c7d6c7ab8c4"}
Jul 16 22:41:17.036762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.036698    2581 scope.go:117] "RemoveContainer" containerID="cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"
Jul 16 22:41:17.045997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.045982    2581 scope.go:117] "RemoveContainer" containerID="69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"
Jul 16 22:41:17.054004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.053985    2581 scope.go:117] "RemoveContainer" containerID="cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"
Jul 16 22:41:17.054263 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:17.054223    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87\": container with ID starting with cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87 not found: ID does not exist" containerID="cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"
Jul 16 22:41:17.054310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.054272    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87"} err="failed to get container status \"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87\": rpc error: code = NotFound desc = could not find container \"cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87\": container with ID starting with cadc76a2c8fa5fbb30f44ed637faae6b67a36b1a2b052e50dd658c212c4ada87 not found: ID does not exist"
Jul 16 22:41:17.054310 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.054289    2581 scope.go:117] "RemoveContainer" containerID="69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"
Jul 16 22:41:17.054509 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:17.054492    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec\": container with ID starting with 69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec not found: ID does not exist" containerID="69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"
Jul 16 22:41:17.054553 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.054515    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec"} err="failed to get container status \"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec\": rpc error: code = NotFound desc = could not find container \"69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec\": container with ID starting with 69bd115f2d0904fd7ce15908993bce50f75cc71aa5d2aa67c5ccf402098242ec not found: ID does not exist"
Jul 16 22:41:17.062584 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.062563    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:41:17.065737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.065717    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-a0061f-openshift-default-86d595549f-2p4js"]
Jul 16 22:41:17.161157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:17.161126    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" path="/var/lib/kubelet/pods/d799c5a7-bb6b-491e-916c-f19d7512841c/volumes"
Jul 16 22:41:24.941392 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:24.941357    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:41:26.155157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:26.155129    2581 scope.go:117] "RemoveContainer" containerID="3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a"
Jul 16 22:41:27.089667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:27.089632    2581 generic.go:356] "Generic (PLEG): container finished" podID="264aa26d-83ea-418b-9df3-ed76291777c0" containerID="3b1ff8b16f00d68ffd74f6a4a91b3b247b947524872f9005b9383aef8d5fbb51" exitCode=6
Jul 16 22:41:27.089822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:27.089705    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" event={"ID":"264aa26d-83ea-418b-9df3-ed76291777c0","Type":"ContainerDied","Data":"3b1ff8b16f00d68ffd74f6a4a91b3b247b947524872f9005b9383aef8d5fbb51"}
Jul 16 22:41:27.089822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:27.089767    2581 scope.go:117] "RemoveContainer" containerID="3e56eb3949619418dc7de8f9875594816442ba6148c382d4a25d271cc523f59a"
Jul 16 22:41:27.090048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:27.090034    2581 scope.go:117] "RemoveContainer" containerID="3b1ff8b16f00d68ffd74f6a4a91b3b247b947524872f9005b9383aef8d5fbb51"
Jul 16 22:41:27.090299 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:27.090277    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw_odh-ai-gateway-infra(264aa26d-83ea-418b-9df3-ed76291777c0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" podUID="264aa26d-83ea-418b-9df3-ed76291777c0"
Jul 16 22:41:28.117712 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:28.117676    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:28.249822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:28.249798    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:28.323437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:28.323404    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca\") pod \"264aa26d-83ea-418b-9df3-ed76291777c0\" (UID: \"264aa26d-83ea-418b-9df3-ed76291777c0\") "
Jul 16 22:41:28.323728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:28.323707    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "264aa26d-83ea-418b-9df3-ed76291777c0" (UID: "264aa26d-83ea-418b-9df3-ed76291777c0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:28.424454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:28.424387    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/264aa26d-83ea-418b-9df3-ed76291777c0-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:29.101298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.101269    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"
Jul 16 22:41:29.101475 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.101256    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw" event={"ID":"264aa26d-83ea-418b-9df3-ed76291777c0","Type":"ContainerDied","Data":"1f59fbca8a68ff1c45c576be9b891e2ae63c05755247c411bb6c12c97a8d0974"}
Jul 16 22:41:29.101475 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.101437    2581 scope.go:117] "RemoveContainer" containerID="3b1ff8b16f00d68ffd74f6a4a91b3b247b947524872f9005b9383aef8d5fbb51"
Jul 16 22:41:29.124287 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.124258    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:29.129539 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.129517    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-7tfxw"]
Jul 16 22:41:29.160213 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:29.160191    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" path="/var/lib/kubelet/pods/264aa26d-83ea-418b-9df3-ed76291777c0/volumes"
Jul 16 22:41:37.269454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:37.269419    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:41:37.269845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:37.269664    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy" containerID="cri-o://ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd" gracePeriod=30
Jul 16 22:41:38.188737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.188700    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:41:38.189176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189161    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="928cf688-cc12-4150-8285-85f2ee813e41" containerName="revoke-keys"
Jul 16 22:41:38.189249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189178    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="928cf688-cc12-4150-8285-85f2ee813e41" containerName="revoke-keys"
Jul 16 22:41:38.189249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189206    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:38.189249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189214    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:38.189249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189241    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:38.189249 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189251    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189266    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189275    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189297    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189305    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189324    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189332    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189402    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189411    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189421    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="928cf688-cc12-4150-8285-85f2ee813e41" containerName="revoke-keys"
Jul 16 22:41:38.189435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.189431    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:38.192594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.192576    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.200406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.200386    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-fbed9b-openshift-default-dockercfg-n6zq5\""
Jul 16 22:41:38.220757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.220734    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:41:38.298468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298433    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298487    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298525    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298546    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298563    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298595    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298617    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298644    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.298902 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.298657    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399429 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399395    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399444    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399471    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399502    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399532    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399592 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399574    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399610    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399654    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399677    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.399875 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399853    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.400098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.399909    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.400155 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.400102    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.400223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.400204    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.400223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.400220    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.401987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.401960    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.402183 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.402165    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.412281 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.412259    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.414378 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.414359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp\") pod \"e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") " pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.502282 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.502185    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:38.640847 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.640823    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:41:38.643413 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:38.643372    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod781cea4b_97d1_4af5_9ee2_6b8e41596578.slice/crio-d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b WatchSource:0}: Error finding container d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b: Status 404 returned error can't find the container with id d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b
Jul 16 22:41:38.645402 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.645364    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:38.645509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.645442    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:38.645509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:38.645483    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:39.160029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:39.159995    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerStarted","Data":"eebe4d24abc66f1bb57d1af607dd5c9c5fc2ca5c471f68418fb1d8b795ceb095"}
Jul 16 22:41:39.160194 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:39.160041    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerStarted","Data":"d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b"}
Jul 16 22:41:39.177933 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:39.177886    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podStartSLOduration=1.177870623 podStartE2EDuration="1.177870623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:39.176920225 +0000 UTC m=+3286.506360151" watchObservedRunningTime="2026-07-16 22:41:39.177870623 +0000 UTC m=+3286.507310525"
Jul 16 22:41:39.502644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:39.502564    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:39.507336 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:39.507309    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:40.161016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:40.160978    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:40.161809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:40.161787    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:42.518304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.518283    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:41:42.637935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.637909    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.637943    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.637983    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638021    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638052    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638120 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638075    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638125    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-25s9h\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638157    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638198    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy\") pod \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\" (UID: \"686a8934-16c8-4e3d-b05c-f2e8fe6c0486\") "
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638209    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.638448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638390    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.638758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638574    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.638758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638587    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:42.638758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638593    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.638758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638637    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.638966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.638776    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data" (OuterVolumeSpecName: "istio-data") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.640433 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.640403    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:41:42.640727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.640706    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token" (OuterVolumeSpecName: "istio-token") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:42.640878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.640861    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h" (OuterVolumeSpecName: "kube-api-access-25s9h") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "kube-api-access-25s9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:41:42.640878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.640865    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "686a8934-16c8-4e3d-b05c-f2e8fe6c0486" (UID: "686a8934-16c8-4e3d-b05c-f2e8fe6c0486"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:41:42.739201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739172    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.739201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739195    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.739201 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739204    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.739440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739212    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.739440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739221    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.739440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:42.739248    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25s9h\" (UniqueName: \"kubernetes.io/projected/686a8934-16c8-4e3d-b05c-f2e8fe6c0486-kube-api-access-25s9h\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:41:43.176157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.176131    2581 generic.go:356] "Generic (PLEG): container finished" podID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerID="ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd" exitCode=0
Jul 16 22:41:43.176300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.176176    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"
Jul 16 22:41:43.176300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.176211    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerDied","Data":"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"}
Jul 16 22:41:43.176300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.176276    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x" event={"ID":"686a8934-16c8-4e3d-b05c-f2e8fe6c0486","Type":"ContainerDied","Data":"5cace5ddd092811b5675bc07a8d6e4b56aba63895918517c6a7d74bce5c0d71a"}
Jul 16 22:41:43.176300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.176299    2581 scope.go:117] "RemoveContainer" containerID="ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"
Jul 16 22:41:43.184760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.184722    2581 scope.go:117] "RemoveContainer" containerID="f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"
Jul 16 22:41:43.193028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.193011    2581 scope.go:117] "RemoveContainer" containerID="ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"
Jul 16 22:41:43.193277 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:43.193259    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd\": container with ID starting with ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd not found: ID does not exist" containerID="ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"
Jul 16 22:41:43.193361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.193290    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd"} err="failed to get container status \"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd\": rpc error: code = NotFound desc = could not find container \"ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd\": container with ID starting with ef68a7ac5bd022e3039680e1a97dc9b0ad8b1e809e44e2cc61ae99e7c998bccd not found: ID does not exist"
Jul 16 22:41:43.193361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.193314    2581 scope.go:117] "RemoveContainer" containerID="f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"
Jul 16 22:41:43.193546 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:43.193530    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085\": container with ID starting with f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085 not found: ID does not exist" containerID="f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"
Jul 16 22:41:43.193607 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.193566    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085"} err="failed to get container status \"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085\": rpc error: code = NotFound desc = could not find container \"f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085\": container with ID starting with f1c9fb998afac69f2893bf8dc6ae27613a62302a7d4de68f9b4d72137f2cd085 not found: ID does not exist"
Jul 16 22:41:43.196552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.196510    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:41:43.198876 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:43.198855    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-24b876-openshift-default-5bddb5b667-5h49x"]
Jul 16 22:41:45.160145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.160114    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" path="/var/lib/kubelet/pods/686a8934-16c8-4e3d-b05c-f2e8fe6c0486/volumes"
Jul 16 22:41:45.975544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975507    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:41:45.975881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975866    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.975931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975883    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.975931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975895    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.975931 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.976024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975952    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.976024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975963    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d799c5a7-bb6b-491e-916c-f19d7512841c" containerName="istio-proxy"
Jul 16 22:41:45.976024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975969    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="264aa26d-83ea-418b-9df3-ed76291777c0" containerName="revoke-keys"
Jul 16 22:41:45.976024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.975975    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="686a8934-16c8-4e3d-b05c-f2e8fe6c0486" containerName="istio-proxy"
Jul 16 22:41:45.980194 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.980177    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:41:45.982895 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.982876    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:41:45.982985 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.982882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:41:45.991114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:45.991088    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:41:46.067194 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.067159    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb\" (UID: \"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:41:46.168461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.168420    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb\" (UID: \"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:41:46.169031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.169009    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb\" (UID: \"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:41:46.275098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.275014    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:41:46.283087 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.283056    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.286501 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.286477    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-106c85-openshift-default-dockercfg-jx6x7\""
Jul 16 22:41:46.291124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.291101    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:41:46.293440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.293414    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:41:46.370662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370631    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.370789 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.370789 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370705    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.370789 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370734    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.371303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370800    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.371303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370836    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.371303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370867    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.371303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370898    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.371303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.370917    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tstqt\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.412709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.412675    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:41:46.441780 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.441754    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:41:46.447852 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:46.447039    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod483d09ac_fd46_45b5_b8f0_7fc2a555dcfb.slice/crio-29270e88446924bcc965ce7d9484d8caf3ada49d28d5b71623ffbcf8e1dc9486 WatchSource:0}: Error finding container 29270e88446924bcc965ce7d9484d8caf3ada49d28d5b71623ffbcf8e1dc9486: Status 404 returned error can't find the container with id 29270e88446924bcc965ce7d9484d8caf3ada49d28d5b71623ffbcf8e1dc9486
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.471971    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472030    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472067    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472111    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472135    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tstqt\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472242    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472295    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472332    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472362    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473029 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.472729    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473633 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.473282    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.473989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.473968    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.474272 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.474192    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.474455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.474433    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.476090 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.476067    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.476365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.476347    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.480477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.480453    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.481716 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.481696    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tstqt\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt\") pod \"e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") " pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.597009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.596978    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:46.745421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.745387    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:41:46.746626 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:46.746601    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c562dd1_ea43_47b8_95af_e20eb2907723.slice/crio-edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110 WatchSource:0}: Error finding container edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110: Status 404 returned error can't find the container with id edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110
Jul 16 22:41:46.749063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.749029    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:46.749173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.749091    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:46.749173 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:46.749120    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:47.198445 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.198405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerStarted","Data":"a803bc7e6cb9f62b581686f52664a23d14ee4a8f730dbd1d795e18f4668d3f18"}
Jul 16 22:41:47.198938 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.198464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerStarted","Data":"edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110"}
Jul 16 22:41:47.200123 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.200099    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerStarted","Data":"da97128ef3a458f747d4973b712f3abba54bc5e1fcff059ee6e77c8d91aae9d1"}
Jul 16 22:41:47.200282 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.200147    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerStarted","Data":"29270e88446924bcc965ce7d9484d8caf3ada49d28d5b71623ffbcf8e1dc9486"}
Jul 16 22:41:47.248364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.248303    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podStartSLOduration=1.248284219 podStartE2EDuration="1.248284219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:47.246655582 +0000 UTC m=+3294.576095484" watchObservedRunningTime="2026-07-16 22:41:47.248284219 +0000 UTC m=+3294.577724123"
Jul 16 22:41:47.268665 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.268619    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" podStartSLOduration=2.26860082 podStartE2EDuration="2.26860082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:47.266806115 +0000 UTC m=+3294.596246017" watchObservedRunningTime="2026-07-16 22:41:47.26860082 +0000 UTC m=+3294.598040724"
Jul 16 22:41:47.597477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.597442    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:47.602525 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:47.602498    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:48.206362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:48.206322    2581 generic.go:356] "Generic (PLEG): container finished" podID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerID="da97128ef3a458f747d4973b712f3abba54bc5e1fcff059ee6e77c8d91aae9d1" exitCode=6
Jul 16 22:41:48.206775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:48.206405    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerDied","Data":"da97128ef3a458f747d4973b712f3abba54bc5e1fcff059ee6e77c8d91aae9d1"}
Jul 16 22:41:48.206775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:48.206551    2581 scope.go:117] "RemoveContainer" containerID="da97128ef3a458f747d4973b712f3abba54bc5e1fcff059ee6e77c8d91aae9d1"
Jul 16 22:41:48.206921 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:48.206871    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:48.207882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:48.207863    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:41:49.212886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:49.212851    2581 generic.go:356] "Generic (PLEG): container finished" podID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerID="ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051" exitCode=6
Jul 16 22:41:49.213355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:49.212918    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerDied","Data":"ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051"}
Jul 16 22:41:49.213355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:49.212973    2581 scope.go:117] "RemoveContainer" containerID="da97128ef3a458f747d4973b712f3abba54bc5e1fcff059ee6e77c8d91aae9d1"
Jul 16 22:41:49.213355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:49.213179    2581 scope.go:117] "RemoveContainer" containerID="ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051"
Jul 16 22:41:49.213512 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:49.213436    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb_odh-ai-gateway-infra(483d09ac-fd46-45b5-b8f0-7fc2a555dcfb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb"
Jul 16 22:41:50.219752 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:50.219713    2581 scope.go:117] "RemoveContainer" containerID="ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051"
Jul 16 22:41:50.220203 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:50.219893    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb_odh-ai-gateway-infra(483d09ac-fd46-45b5-b8f0-7fc2a555dcfb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb"
Jul 16 22:41:54.538464 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:54.538389    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:41:54.568916 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:54.568847    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:41:55.112569 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.112532    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:41:55.162833 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.162794    2581 patch_prober.go:28] interesting pod/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Jul 16 22:41:55.162992 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.162852    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Jul 16 22:41:55.246490 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.246452    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:41:55.247387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.247347    2581 generic.go:356] "Generic (PLEG): container finished" podID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerID="eebe4d24abc66f1bb57d1af607dd5c9c5fc2ca5c471f68418fb1d8b795ceb095" exitCode=0
Jul 16 22:41:55.249823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.249803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerDied","Data":"eebe4d24abc66f1bb57d1af607dd5c9c5fc2ca5c471f68418fb1d8b795ceb095"}
Jul 16 22:41:55.249910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.249899    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:41:55.250218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.250198    2581 scope.go:117] "RemoveContainer" containerID="eebe4d24abc66f1bb57d1af607dd5c9c5fc2ca5c471f68418fb1d8b795ceb095"
Jul 16 22:41:55.251288 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.251262    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:55.251367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.251338    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:55.251406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.251396    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:41:55.257181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.257159    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:41:55.352252 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.352187    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9\" (UID: \"c27e49c5-9374-41bb-95dd-9f1f3d943adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:41:55.453628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.453535    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9\" (UID: \"c27e49c5-9374-41bb-95dd-9f1f3d943adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:41:55.454133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.454112    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9\" (UID: \"c27e49c5-9374-41bb-95dd-9f1f3d943adf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:41:55.562665 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.562633    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:41:55.696559 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:55.696529    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:41:55.698075 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:41:55.698043    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc27e49c5_9374_41bb_95dd_9f1f3d943adf.slice/crio-65519c3722fa31bec2edfde73a874892147e76da07e3e3bda80b23b637850f84 WatchSource:0}: Error finding container 65519c3722fa31bec2edfde73a874892147e76da07e3e3bda80b23b637850f84: Status 404 returned error can't find the container with id 65519c3722fa31bec2edfde73a874892147e76da07e3e3bda80b23b637850f84
Jul 16 22:41:56.258331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.257350    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerStarted","Data":"05496b8116e703d11d34a0ab39b433e10c01ae88e23f5dbdb29a01aed78b88b5"}
Jul 16 22:41:56.267423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.265649    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerStarted","Data":"61ff0827f0baf10e73e02ba62cbfca02717007e0352646c525c393eec005192e"}
Jul 16 22:41:56.267423 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.265717    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerStarted","Data":"65519c3722fa31bec2edfde73a874892147e76da07e3e3bda80b23b637850f84"}
Jul 16 22:41:56.306265 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.306191    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" podStartSLOduration=1.306173925 podStartE2EDuration="1.306173925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:56.304944115 +0000 UTC m=+3303.634384018" watchObservedRunningTime="2026-07-16 22:41:56.306173925 +0000 UTC m=+3303.635613827"
Jul 16 22:41:56.503160 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.503120    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:41:56.503980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.503955    2581 patch_prober.go:28] interesting pod/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Jul 16 22:41:56.504032 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:56.504009    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Jul 16 22:41:57.278531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:57.278492    2581 generic.go:356] "Generic (PLEG): container finished" podID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerID="61ff0827f0baf10e73e02ba62cbfca02717007e0352646c525c393eec005192e" exitCode=6
Jul 16 22:41:57.279317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:57.279279    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerDied","Data":"61ff0827f0baf10e73e02ba62cbfca02717007e0352646c525c393eec005192e"}
Jul 16 22:41:57.279472 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:57.279412    2581 scope.go:117] "RemoveContainer" containerID="61ff0827f0baf10e73e02ba62cbfca02717007e0352646c525c393eec005192e"
Jul 16 22:41:57.503443 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:57.503404    2581 patch_prober.go:28] interesting pod/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Jul 16 22:41:57.503624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:57.503458    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Jul 16 22:41:58.301262 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.300633    2581 generic.go:356] "Generic (PLEG): container finished" podID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerID="67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce" exitCode=6
Jul 16 22:41:58.301262 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.300770    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerDied","Data":"67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce"}
Jul 16 22:41:58.301262 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.300828    2581 scope.go:117] "RemoveContainer" containerID="61ff0827f0baf10e73e02ba62cbfca02717007e0352646c525c393eec005192e"
Jul 16 22:41:58.301867 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.301472    2581 scope.go:117] "RemoveContainer" containerID="67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce"
Jul 16 22:41:58.301867 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:58.301842    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9_odh-ai-gateway-infra(c27e49c5-9374-41bb-95dd-9f1f3d943adf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf"
Jul 16 22:41:58.503513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.503412    2581 patch_prober.go:28] interesting pod/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Jul 16 22:41:58.503513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:58.503479    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Jul 16 22:41:59.307447 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:59.307417    2581 scope.go:117] "RemoveContainer" containerID="67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce"
Jul 16 22:41:59.307812 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:41:59.307615    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9_odh-ai-gateway-infra(c27e49c5-9374-41bb-95dd-9f1f3d943adf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf"
Jul 16 22:41:59.503531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:59.503491    2581 patch_prober.go:28] interesting pod/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused" start-of-body=
Jul 16 22:41:59.503739 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:41:59.503553    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.123:15021/healthz/ready\": dial tcp 10.133.0.123:15021: connect: connection refused"
Jul 16 22:42:00.506957 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:00.506927    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:42:00.507365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:00.507338    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:42:00.508145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:00.508128    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:42:03.208088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.208048    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Jul 16 22:42:03.208511 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.208103    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Jul 16 22:42:03.329957 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.329874    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerID="a803bc7e6cb9f62b581686f52664a23d14ee4a8f730dbd1d795e18f4668d3f18" exitCode=0
Jul 16 22:42:03.329957 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.329924    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerDied","Data":"a803bc7e6cb9f62b581686f52664a23d14ee4a8f730dbd1d795e18f4668d3f18"}
Jul 16 22:42:03.330335 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.330295    2581 scope.go:117] "RemoveContainer" containerID="a803bc7e6cb9f62b581686f52664a23d14ee4a8f730dbd1d795e18f4668d3f18"
Jul 16 22:42:03.331289 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.331263    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:42:03.331396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.331324    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:42:03.331396 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:03.331363    2581 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 22:42:04.154641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:04.154597    2581 scope.go:117] "RemoveContainer" containerID="ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051"
Jul 16 22:42:04.336647 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:04.336613    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerStarted","Data":"47effa85f305af135d17a2db7924b40d86ec04be12fb35fd9e37005ea356c7af"}
Jul 16 22:42:04.597853 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:04.597811    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:42:04.598059 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:04.598037    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Jul 16 22:42:04.598124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:04.598083    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Jul 16 22:42:05.342566 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.342536    2581 generic.go:356] "Generic (PLEG): container finished" podID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerID="1110716c416fa30edf4ec29e2e852a6fa587e446790db87c55e1241122c154f7" exitCode=6
Jul 16 22:42:05.342982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.342610    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerDied","Data":"1110716c416fa30edf4ec29e2e852a6fa587e446790db87c55e1241122c154f7"}
Jul 16 22:42:05.342982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.342670    2581 scope.go:117] "RemoveContainer" containerID="ea8a1cf4a3ce7b6a650c09e08176b6c234fb42d39d9b428f13859dabfc5ad051"
Jul 16 22:42:05.342982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.342933    2581 scope.go:117] "RemoveContainer" containerID="1110716c416fa30edf4ec29e2e852a6fa587e446790db87c55e1241122c154f7"
Jul 16 22:42:05.343174 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:42:05.343156    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb_odh-ai-gateway-infra(483d09ac-fd46-45b5-b8f0-7fc2a555dcfb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb"
Jul 16 22:42:05.597947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.597866    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Jul 16 22:42:05.597947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:05.597920    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Jul 16 22:42:06.399567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.399532    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:42:06.565920 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.565640    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:42:06.597510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.597479    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Jul 16 22:42:06.597634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.597531    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Jul 16 22:42:06.671723 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.671632    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca\") pod \"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb\" (UID: \"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb\") "
Jul 16 22:42:06.672002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.671976    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" (UID: "483d09ac-fd46-45b5-b8f0-7fc2a555dcfb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:06.772753 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:06.772713    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:42:07.361905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.361857    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb" event={"ID":"483d09ac-fd46-45b5-b8f0-7fc2a555dcfb","Type":"ContainerDied","Data":"29270e88446924bcc965ce7d9484d8caf3ada49d28d5b71623ffbcf8e1dc9486"}
Jul 16 22:42:07.362124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.361873    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"
Jul 16 22:42:07.362124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.361976    2581 scope.go:117] "RemoveContainer" containerID="1110716c416fa30edf4ec29e2e852a6fa587e446790db87c55e1241122c154f7"
Jul 16 22:42:07.384790 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.384762    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:42:07.388507 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.388481    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-s9phb"]
Jul 16 22:42:07.597544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.597513    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused" start-of-body=
Jul 16 22:42:07.597937 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:07.597571    2581 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.125:15021/healthz/ready\": dial tcp 10.133.0.125:15021: connect: connection refused"
Jul 16 22:42:08.601793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:08.601761    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:42:08.602272 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:08.602030    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:42:08.602832 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:08.602817    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:42:09.160597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.160561    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" path="/var/lib/kubelet/pods/483d09ac-fd46-45b5-b8f0-7fc2a555dcfb/volumes"
Jul 16 22:42:09.311195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311163    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:42:09.311537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311523    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.311588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311539    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.311588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.311588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311568    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.311682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311617    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.311682 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.311624    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:09.315904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.315886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.318304 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.318281    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"default-dockercfg-cjhlj\""
Jul 16 22:42:09.319357 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.319329    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"auth-test-model-fbed9b-kserve-self-signed-certs\""
Jul 16 22:42:09.319498 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.319331    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"openshift-service-ca.crt\""
Jul 16 22:42:09.319616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.319359    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-fbed9b\"/\"kube-root-ca.crt\""
Jul 16 22:42:09.324744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.324720    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:42:09.496030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.495936    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.496030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.495976    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhxmr\" (UniqueName: \"kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.496030 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.496003    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.496303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.496176    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.496303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.496280    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.496385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.496316    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597579 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597545    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597598    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597659    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597680    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhxmr\" (UniqueName: \"kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597862    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.597966 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597944    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.598019 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.597956    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.598167 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.598146    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.598294 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.598276    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.600652 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.600632    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.600834 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.600815    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.612840 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.612810    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhxmr\" (UniqueName: \"kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr\") pod \"auth-test-model-fbed9b-kserve-666544ccc9-hvd5j\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") " pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.626704 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.626679    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:09.774638 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:09.774491    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:42:09.777727 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:42:09.777679    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfa949b46_fa04_4418_8ae8_3dbe083b83e5.slice/crio-b529bf7bce73c7acbd71ac3a3e0dc25d5df623971c0536be56f4b70483971c7c WatchSource:0}: Error finding container b529bf7bce73c7acbd71ac3a3e0dc25d5df623971c0536be56f4b70483971c7c: Status 404 returned error can't find the container with id b529bf7bce73c7acbd71ac3a3e0dc25d5df623971c0536be56f4b70483971c7c
Jul 16 22:42:10.377332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:10.377292    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" event={"ID":"fa949b46-fa04-4418-8ae8-3dbe083b83e5","Type":"ContainerStarted","Data":"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"}
Jul 16 22:42:10.377332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:10.377338    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" event={"ID":"fa949b46-fa04-4418-8ae8-3dbe083b83e5","Type":"ContainerStarted","Data":"b529bf7bce73c7acbd71ac3a3e0dc25d5df623971c0536be56f4b70483971c7c"}
Jul 16 22:42:10.399514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:10.399462    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" podStartSLOduration=1.399443735 podStartE2EDuration="1.399443735s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:10.397855606 +0000 UTC m=+3317.727295510" watchObservedRunningTime="2026-07-16 22:42:10.399443735 +0000 UTC m=+3317.728883638"
Jul 16 22:42:11.155297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:11.155264    2581 scope.go:117] "RemoveContainer" containerID="67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce"
Jul 16 22:42:12.390988 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:12.390953    2581 generic.go:356] "Generic (PLEG): container finished" podID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerID="8f41f623073599f7288cd5d840fd6cd46784f9197b0a587e90ed80bdfcf801b7" exitCode=6
Jul 16 22:42:12.391425 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:12.391003    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerDied","Data":"8f41f623073599f7288cd5d840fd6cd46784f9197b0a587e90ed80bdfcf801b7"}
Jul 16 22:42:12.391425 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:12.391074    2581 scope.go:117] "RemoveContainer" containerID="67f68d9302d86c184f02d57f84b09143510e373580222d2019a1f41bfba0cdce"
Jul 16 22:42:12.391425 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:12.391419    2581 scope.go:117] "RemoveContainer" containerID="8f41f623073599f7288cd5d840fd6cd46784f9197b0a587e90ed80bdfcf801b7"
Jul 16 22:42:12.391675 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:42:12.391650    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9_odh-ai-gateway-infra(c27e49c5-9374-41bb-95dd-9f1f3d943adf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf"
Jul 16 22:42:12.711325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:12.711240    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:42:13.419764 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:13.419723    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:42:13.557516 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:13.557494    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:42:13.636588 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:13.636554    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca\") pod \"c27e49c5-9374-41bb-95dd-9f1f3d943adf\" (UID: \"c27e49c5-9374-41bb-95dd-9f1f3d943adf\") "
Jul 16 22:42:13.636927 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:13.636902    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c27e49c5-9374-41bb-95dd-9f1f3d943adf" (UID: "c27e49c5-9374-41bb-95dd-9f1f3d943adf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:13.737979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:13.737896    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c27e49c5-9374-41bb-95dd-9f1f3d943adf-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:42:14.402301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:14.402205    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"
Jul 16 22:42:14.402301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:14.402220    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9" event={"ID":"c27e49c5-9374-41bb-95dd-9f1f3d943adf","Type":"ContainerDied","Data":"65519c3722fa31bec2edfde73a874892147e76da07e3e3bda80b23b637850f84"}
Jul 16 22:42:14.402549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:14.402311    2581 scope.go:117] "RemoveContainer" containerID="8f41f623073599f7288cd5d840fd6cd46784f9197b0a587e90ed80bdfcf801b7"
Jul 16 22:42:14.422481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:14.422451    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:42:14.426809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:14.426772    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-8q4b9"]
Jul 16 22:42:15.160046 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:15.160013    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" path="/var/lib/kubelet/pods/c27e49c5-9374-41bb-95dd-9f1f3d943adf/volumes"
Jul 16 22:42:19.626916 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:19.626872    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:19.627400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:19.626940    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:19.639288 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:19.639263    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:20.442728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:20.442697    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:42:34.230044 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:34.230007    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:42:45.297497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297464    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297774    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297783    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297793    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297798    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297810    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297816    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297821    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297826    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.297878 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297881    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.298166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297890    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.298166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297895    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="c27e49c5-9374-41bb-95dd-9f1f3d943adf" containerName="revoke-keys"
Jul 16 22:42:45.298166 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.297902    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="483d09ac-fd46-45b5-b8f0-7fc2a555dcfb" containerName="revoke-keys"
Jul 16 22:42:45.300654 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.300634    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:42:45.302998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.302979    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:42:45.303093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.303005    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:42:45.306872 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.306850    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:42:45.423190 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.423155    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j\" (UID: \"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:42:45.524481 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.524436    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j\" (UID: \"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:42:45.525022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.525004    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j\" (UID: \"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:42:45.612451 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.612424    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:42:45.965582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:45.965554    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:42:45.966884 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:42:45.966856    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad9afc1_9eb4_4b66_9ea1_df3b2591a398.slice/crio-fa1d5779d9854d44070803a4dc9142e1880c77d80b3df997a788a523fe147e35 WatchSource:0}: Error finding container fa1d5779d9854d44070803a4dc9142e1880c77d80b3df997a788a523fe147e35: Status 404 returned error can't find the container with id fa1d5779d9854d44070803a4dc9142e1880c77d80b3df997a788a523fe147e35
Jul 16 22:42:46.553466 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:46.553374    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerStarted","Data":"2f7c6388db281b46d6d46ed2e576b30192e97b014b903b1037200a248176953b"}
Jul 16 22:42:46.553466 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:46.553450    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerStarted","Data":"fa1d5779d9854d44070803a4dc9142e1880c77d80b3df997a788a523fe147e35"}
Jul 16 22:42:46.573290 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:46.573222    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" podStartSLOduration=1.5732088819999999 podStartE2EDuration="1.573208882s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:46.57014489 +0000 UTC m=+3353.899584790" watchObservedRunningTime="2026-07-16 22:42:46.573208882 +0000 UTC m=+3353.902648784"
Jul 16 22:42:47.559467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:47.559432    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerID="2f7c6388db281b46d6d46ed2e576b30192e97b014b903b1037200a248176953b" exitCode=6
Jul 16 22:42:47.559868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:47.559523    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerDied","Data":"2f7c6388db281b46d6d46ed2e576b30192e97b014b903b1037200a248176953b"}
Jul 16 22:42:47.559868 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:47.559688    2581 scope.go:117] "RemoveContainer" containerID="2f7c6388db281b46d6d46ed2e576b30192e97b014b903b1037200a248176953b"
Jul 16 22:42:48.566159 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:48.566076    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerID="b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c" exitCode=6
Jul 16 22:42:48.566159 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:48.566141    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerDied","Data":"b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c"}
Jul 16 22:42:48.566630 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:48.566189    2581 scope.go:117] "RemoveContainer" containerID="2f7c6388db281b46d6d46ed2e576b30192e97b014b903b1037200a248176953b"
Jul 16 22:42:48.566630 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:48.566356    2581 scope.go:117] "RemoveContainer" containerID="b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c"
Jul 16 22:42:48.566630 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:42:48.566592    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j_odh-ai-gateway-infra(7ad9afc1-9eb4-4b66-9ea1-df3b2591a398)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398"
Jul 16 22:42:49.573474 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:49.573443    2581 scope.go:117] "RemoveContainer" containerID="b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c"
Jul 16 22:42:49.573857 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:42:49.573646    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j_odh-ai-gateway-infra(7ad9afc1-9eb4-4b66-9ea1-df3b2591a398)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398"
Jul 16 22:42:50.545936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:42:50.545892    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:43:00.155496 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:00.155463    2581 scope.go:117] "RemoveContainer" containerID="b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c"
Jul 16 22:43:00.635456 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:00.635418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerStarted","Data":"bddfe43e5df0e4aa3fd20f732274b43d057029186e84cac9b359ef12496abb23"}
Jul 16 22:43:01.641801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:01.641765    2581 generic.go:356] "Generic (PLEG): container finished" podID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerID="bddfe43e5df0e4aa3fd20f732274b43d057029186e84cac9b359ef12496abb23" exitCode=6
Jul 16 22:43:01.642259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:01.641839    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerDied","Data":"bddfe43e5df0e4aa3fd20f732274b43d057029186e84cac9b359ef12496abb23"}
Jul 16 22:43:01.642259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:01.641911    2581 scope.go:117] "RemoveContainer" containerID="b3df3d93bfea0628d63494518c8bbaed653a2c025594cd5213ca12db7ff2cd2c"
Jul 16 22:43:01.642259 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:01.642180    2581 scope.go:117] "RemoveContainer" containerID="bddfe43e5df0e4aa3fd20f732274b43d057029186e84cac9b359ef12496abb23"
Jul 16 22:43:01.642477 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:43:01.642424    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j_odh-ai-gateway-infra(7ad9afc1-9eb4-4b66-9ea1-df3b2591a398)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398"
Jul 16 22:43:01.672075 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:01.672044    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:43:02.779283 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:02.779261    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:43:02.858762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:02.858724    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca\") pod \"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398\" (UID: \"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398\") "
Jul 16 22:43:02.859075 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:02.859053    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" (UID: "7ad9afc1-9eb4-4b66-9ea1-df3b2591a398"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:02.960353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:02.960271    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:43:03.653361 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:03.653336    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"
Jul 16 22:43:03.653545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:03.653332    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j" event={"ID":"7ad9afc1-9eb4-4b66-9ea1-df3b2591a398","Type":"ContainerDied","Data":"fa1d5779d9854d44070803a4dc9142e1880c77d80b3df997a788a523fe147e35"}
Jul 16 22:43:03.653545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:03.653495    2581 scope.go:117] "RemoveContainer" containerID="bddfe43e5df0e4aa3fd20f732274b43d057029186e84cac9b359ef12496abb23"
Jul 16 22:43:03.671953 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:03.671923    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:43:03.676243 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:03.676208    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-gvg7j"]
Jul 16 22:43:05.159827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:05.159792    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" path="/var/lib/kubelet/pods/7ad9afc1-9eb4-4b66-9ea1-df3b2591a398/volumes"
Jul 16 22:43:09.607814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:09.607782    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:43:25.213552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:25.213519    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:43:35.336904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.336866    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:35.337409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337343    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337361    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337391    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337398    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337407    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337416    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337509    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337522    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.337672 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.337533    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad9afc1-9eb4-4b66-9ea1-df3b2591a398" containerName="revoke-keys"
Jul 16 22:43:35.340391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.340371    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:35.342899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.342882    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:43:35.342988 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.342964    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:43:35.348475 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.348449    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:35.447035 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.446993    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2\" (UID: \"31abf88c-1750-4afa-8ec1-6a2bbe64188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:35.547561 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.547519    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2\" (UID: \"31abf88c-1750-4afa-8ec1-6a2bbe64188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:35.548158 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.548135    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2\" (UID: \"31abf88c-1750-4afa-8ec1-6a2bbe64188e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:35.650944 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.650851    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:35.793209 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.793186    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:35.794906 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:43:35.794872    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31abf88c_1750_4afa_8ec1_6a2bbe64188e.slice/crio-4330dfe7baddfa92ee457507a3711c3fbac5b9e1bd50af257046850079964845 WatchSource:0}: Error finding container 4330dfe7baddfa92ee457507a3711c3fbac5b9e1bd50af257046850079964845: Status 404 returned error can't find the container with id 4330dfe7baddfa92ee457507a3711c3fbac5b9e1bd50af257046850079964845
Jul 16 22:43:35.806526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:35.806503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" event={"ID":"31abf88c-1750-4afa-8ec1-6a2bbe64188e","Type":"ContainerStarted","Data":"4330dfe7baddfa92ee457507a3711c3fbac5b9e1bd50af257046850079964845"}
Jul 16 22:43:36.813504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:36.813410    2581 generic.go:356] "Generic (PLEG): container finished" podID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerID="c33fbda01c454327705213d1dd6bcbe1707d9aab8b978de965f9af02e3286577" exitCode=6
Jul 16 22:43:36.813504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:36.813479    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" event={"ID":"31abf88c-1750-4afa-8ec1-6a2bbe64188e","Type":"ContainerDied","Data":"c33fbda01c454327705213d1dd6bcbe1707d9aab8b978de965f9af02e3286577"}
Jul 16 22:43:36.813905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:36.813643    2581 scope.go:117] "RemoveContainer" containerID="c33fbda01c454327705213d1dd6bcbe1707d9aab8b978de965f9af02e3286577"
Jul 16 22:43:37.819881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:37.819790    2581 generic.go:356] "Generic (PLEG): container finished" podID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerID="861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656" exitCode=6
Jul 16 22:43:37.820321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:37.819871    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" event={"ID":"31abf88c-1750-4afa-8ec1-6a2bbe64188e","Type":"ContainerDied","Data":"861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656"}
Jul 16 22:43:37.820321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:37.819932    2581 scope.go:117] "RemoveContainer" containerID="c33fbda01c454327705213d1dd6bcbe1707d9aab8b978de965f9af02e3286577"
Jul 16 22:43:37.820321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:37.820153    2581 scope.go:117] "RemoveContainer" containerID="861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656"
Jul 16 22:43:37.820448 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:43:37.820378    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2_odh-ai-gateway-infra(31abf88c-1750-4afa-8ec1-6a2bbe64188e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e"
Jul 16 22:43:38.827052 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:38.827022    2581 scope.go:117] "RemoveContainer" containerID="861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656"
Jul 16 22:43:38.827504 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:43:38.827293    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2_odh-ai-gateway-infra(31abf88c-1750-4afa-8ec1-6a2bbe64188e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e"
Jul 16 22:43:44.212107 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:44.212060    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:43:52.155421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:52.155391    2581 scope.go:117] "RemoveContainer" containerID="861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656"
Jul 16 22:43:52.893862 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:52.893828    2581 generic.go:356] "Generic (PLEG): container finished" podID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerID="8bc6d06fd7e17464d6f476453ae9385cb0ab9940780f1e574babc8eb64baae1e" exitCode=6
Jul 16 22:43:52.894061 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:52.893907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" event={"ID":"31abf88c-1750-4afa-8ec1-6a2bbe64188e","Type":"ContainerDied","Data":"8bc6d06fd7e17464d6f476453ae9385cb0ab9940780f1e574babc8eb64baae1e"}
Jul 16 22:43:52.894061 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:52.893983    2581 scope.go:117] "RemoveContainer" containerID="861b5ef0c224a4474dc2dd11f6fabe03da01a435aa0c4cb7c9ceeab5c6c6d656"
Jul 16 22:43:52.894350 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:52.894324    2581 scope.go:117] "RemoveContainer" containerID="8bc6d06fd7e17464d6f476453ae9385cb0ab9940780f1e574babc8eb64baae1e"
Jul 16 22:43:52.894553 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:43:52.894530    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2_odh-ai-gateway-infra(31abf88c-1750-4afa-8ec1-6a2bbe64188e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e"
Jul 16 22:43:53.919453 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:53.919417    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:54.058990 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.058965    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:54.100150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.100118    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca\") pod \"31abf88c-1750-4afa-8ec1-6a2bbe64188e\" (UID: \"31abf88c-1750-4afa-8ec1-6a2bbe64188e\") "
Jul 16 22:43:54.100458 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.100436    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "31abf88c-1750-4afa-8ec1-6a2bbe64188e" (UID: "31abf88c-1750-4afa-8ec1-6a2bbe64188e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:54.200783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.200746    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/31abf88c-1750-4afa-8ec1-6a2bbe64188e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:43:54.906367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.906342    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"
Jul 16 22:43:54.906367 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.906346    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2" event={"ID":"31abf88c-1750-4afa-8ec1-6a2bbe64188e","Type":"ContainerDied","Data":"4330dfe7baddfa92ee457507a3711c3fbac5b9e1bd50af257046850079964845"}
Jul 16 22:43:54.906640 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.906411    2581 scope.go:117] "RemoveContainer" containerID="8bc6d06fd7e17464d6f476453ae9385cb0ab9940780f1e574babc8eb64baae1e"
Jul 16 22:43:54.929855 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.929831    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:54.934459 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:54.934436    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-rlqd2"]
Jul 16 22:43:55.160506 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:55.160419    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" path="/var/lib/kubelet/pods/31abf88c-1750-4afa-8ec1-6a2bbe64188e/volumes"
Jul 16 22:43:57.661855 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.661824    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662164    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662174    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662181    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662192    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662197    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662259    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662267    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.662345 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.662273    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="31abf88c-1750-4afa-8ec1-6a2bbe64188e" containerName="revoke-keys"
Jul 16 22:43:57.666260 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.666219    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:43:57.668697 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.668671    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:43:57.668810 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.668720    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:43:57.673690 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.673667    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:43:57.725703 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.725668    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj\" (UID: \"61c56225-17fa-4eb9-a5e3-e0cb125b7cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:43:57.826835 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.826785    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj\" (UID: \"61c56225-17fa-4eb9-a5e3-e0cb125b7cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:43:57.827439 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.827422    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj\" (UID: \"61c56225-17fa-4eb9-a5e3-e0cb125b7cff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:43:57.977979 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:57.977886    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:43:58.128374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:58.128348    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:43:58.130406 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:43:58.130381    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61c56225_17fa_4eb9_a5e3_e0cb125b7cff.slice/crio-889fa454ec1014da978388677ac0cdadaa26bd9c83d79bca442ec49dffd45381 WatchSource:0}: Error finding container 889fa454ec1014da978388677ac0cdadaa26bd9c83d79bca442ec49dffd45381: Status 404 returned error can't find the container with id 889fa454ec1014da978388677ac0cdadaa26bd9c83d79bca442ec49dffd45381
Jul 16 22:43:58.927601 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:58.927569    2581 generic.go:356] "Generic (PLEG): container finished" podID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerID="2288779ff11ff3eeeb41aa93822af4a240bb3c9098d7779d8070f5a40e5996b3" exitCode=6
Jul 16 22:43:58.928016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:58.927667    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" event={"ID":"61c56225-17fa-4eb9-a5e3-e0cb125b7cff","Type":"ContainerDied","Data":"2288779ff11ff3eeeb41aa93822af4a240bb3c9098d7779d8070f5a40e5996b3"}
Jul 16 22:43:58.928016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:58.927727    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" event={"ID":"61c56225-17fa-4eb9-a5e3-e0cb125b7cff","Type":"ContainerStarted","Data":"889fa454ec1014da978388677ac0cdadaa26bd9c83d79bca442ec49dffd45381"}
Jul 16 22:43:58.928016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:58.927811    2581 scope.go:117] "RemoveContainer" containerID="2288779ff11ff3eeeb41aa93822af4a240bb3c9098d7779d8070f5a40e5996b3"
Jul 16 22:43:59.934521 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:59.934478    2581 generic.go:356] "Generic (PLEG): container finished" podID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerID="186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc" exitCode=6
Jul 16 22:43:59.934930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:59.934569    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" event={"ID":"61c56225-17fa-4eb9-a5e3-e0cb125b7cff","Type":"ContainerDied","Data":"186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc"}
Jul 16 22:43:59.934930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:59.934661    2581 scope.go:117] "RemoveContainer" containerID="2288779ff11ff3eeeb41aa93822af4a240bb3c9098d7779d8070f5a40e5996b3"
Jul 16 22:43:59.934930 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:43:59.934853    2581 scope.go:117] "RemoveContainer" containerID="186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc"
Jul 16 22:43:59.935089 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:43:59.935071    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj_odh-ai-gateway-infra(61c56225-17fa-4eb9-a5e3-e0cb125b7cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff"
Jul 16 22:44:00.940665 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:00.940635    2581 scope.go:117] "RemoveContainer" containerID="186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc"
Jul 16 22:44:00.941151 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:00.940894    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj_odh-ai-gateway-infra(61c56225-17fa-4eb9-a5e3-e0cb125b7cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff"
Jul 16 22:44:03.519353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:03.519315    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:44:12.154901 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:12.154870    2581 scope.go:117] "RemoveContainer" containerID="186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc"
Jul 16 22:44:13.004570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:13.004489    2581 generic.go:356] "Generic (PLEG): container finished" podID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerID="3efd6b330b705aa1c96b63eb75d02de30690198300b5c26c5aeec8fd948b0604" exitCode=6
Jul 16 22:44:13.004570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:13.004528    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" event={"ID":"61c56225-17fa-4eb9-a5e3-e0cb125b7cff","Type":"ContainerDied","Data":"3efd6b330b705aa1c96b63eb75d02de30690198300b5c26c5aeec8fd948b0604"}
Jul 16 22:44:13.004763 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:13.004577    2581 scope.go:117] "RemoveContainer" containerID="186f913660bea6895760aaea008b9f9e1890d2f444b7f7c069dfbc6e0eecdcfc"
Jul 16 22:44:13.004874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:13.004855    2581 scope.go:117] "RemoveContainer" containerID="3efd6b330b705aa1c96b63eb75d02de30690198300b5c26c5aeec8fd948b0604"
Jul 16 22:44:13.005114 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:13.005092    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj_odh-ai-gateway-infra(61c56225-17fa-4eb9-a5e3-e0cb125b7cff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff"
Jul 16 22:44:14.048775 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:14.048743    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:44:14.189114 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:14.189086    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:44:14.276620 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:14.276581    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca\") pod \"61c56225-17fa-4eb9-a5e3-e0cb125b7cff\" (UID: \"61c56225-17fa-4eb9-a5e3-e0cb125b7cff\") "
Jul 16 22:44:14.276943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:14.276921    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "61c56225-17fa-4eb9-a5e3-e0cb125b7cff" (UID: "61c56225-17fa-4eb9-a5e3-e0cb125b7cff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:14.377820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:14.377781    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/61c56225-17fa-4eb9-a5e3-e0cb125b7cff-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:15.016927 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.016891    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj" event={"ID":"61c56225-17fa-4eb9-a5e3-e0cb125b7cff","Type":"ContainerDied","Data":"889fa454ec1014da978388677ac0cdadaa26bd9c83d79bca442ec49dffd45381"}
Jul 16 22:44:15.017136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.016959    2581 scope.go:117] "RemoveContainer" containerID="3efd6b330b705aa1c96b63eb75d02de30690198300b5c26c5aeec8fd948b0604"
Jul 16 22:44:15.017136 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.016901    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"
Jul 16 22:44:15.053958 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.053928    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:44:15.059471 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.059446    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-v4nrj"]
Jul 16 22:44:15.160149 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:15.160104    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" path="/var/lib/kubelet/pods/61c56225-17fa-4eb9-a5e3-e0cb125b7cff/volumes"
Jul 16 22:44:23.208827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:23.208783    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:44:36.204558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:36.204521    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:44:41.525737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:41.525704    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:44:46.184556 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184516    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184871    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184896    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184901    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184910    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.184943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184917    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.185145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184973    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.185145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184981    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.185145 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.184987    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="61c56225-17fa-4eb9-a5e3-e0cb125b7cff" containerName="revoke-keys"
Jul 16 22:44:46.188069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.188051    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:44:46.190884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.190860    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:44:46.190994 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.190928    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:44:46.197021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.196999    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:44:46.354448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.354412    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w\" (UID: \"3f3b309f-2161-4a53-b674-3541b62501e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:44:46.455462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.455359    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w\" (UID: \"3f3b309f-2161-4a53-b674-3541b62501e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:44:46.456027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.456008    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w\" (UID: \"3f3b309f-2161-4a53-b674-3541b62501e4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:44:46.499164 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.499134    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:44:46.647280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:46.647250    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:44:46.649879 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:44:46.649845    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f3b309f_2161_4a53_b674_3541b62501e4.slice/crio-9569ebe0f794f8c0b6a06372db68157db27b1339d74e3490ad356fb2aa51c50e WatchSource:0}: Error finding container 9569ebe0f794f8c0b6a06372db68157db27b1339d74e3490ad356fb2aa51c50e: Status 404 returned error can't find the container with id 9569ebe0f794f8c0b6a06372db68157db27b1339d74e3490ad356fb2aa51c50e
Jul 16 22:44:47.170255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.170144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerStarted","Data":"96fe74c4492b420faf5b8528fbb5b375c1b029f15c94dc34af637cf78c6455f8"}
Jul 16 22:44:47.170255 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.170198    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerStarted","Data":"9569ebe0f794f8c0b6a06372db68157db27b1339d74e3490ad356fb2aa51c50e"}
Jul 16 22:44:47.201147 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.201093    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" podStartSLOduration=1.201076384 podStartE2EDuration="1.201076384s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:44:47.200801024 +0000 UTC m=+3474.530240927" watchObservedRunningTime="2026-07-16 22:44:47.201076384 +0000 UTC m=+3474.530516341"
Jul 16 22:44:47.896461 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.896419    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:44:47.899826 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.899809    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:44:47.937715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.937684    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:44:47.968649 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:47.968620    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5\" (UID: \"9b72a648-7328-434f-aabe-b5cc626bac96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:44:48.069121 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.069084    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5\" (UID: \"9b72a648-7328-434f-aabe-b5cc626bac96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:44:48.069725 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.069706    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5\" (UID: \"9b72a648-7328-434f-aabe-b5cc626bac96\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:44:48.176355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.176268    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f3b309f-2161-4a53-b674-3541b62501e4" containerID="96fe74c4492b420faf5b8528fbb5b375c1b029f15c94dc34af637cf78c6455f8" exitCode=6
Jul 16 22:44:48.176355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.176334    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerDied","Data":"96fe74c4492b420faf5b8528fbb5b375c1b029f15c94dc34af637cf78c6455f8"}
Jul 16 22:44:48.176626 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.176608    2581 scope.go:117] "RemoveContainer" containerID="96fe74c4492b420faf5b8528fbb5b375c1b029f15c94dc34af637cf78c6455f8"
Jul 16 22:44:48.210041 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.210018    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:44:48.367880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:48.367857    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:44:48.369199 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:44:48.369169    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b72a648_7328_434f_aabe_b5cc626bac96.slice/crio-c889e7556be99678f3ea546b396f8bd91f4b165f5f5042dabdc5c59c07f63936 WatchSource:0}: Error finding container c889e7556be99678f3ea546b396f8bd91f4b165f5f5042dabdc5c59c07f63936: Status 404 returned error can't find the container with id c889e7556be99678f3ea546b396f8bd91f4b165f5f5042dabdc5c59c07f63936
Jul 16 22:44:49.182597 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.182555    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f3b309f-2161-4a53-b674-3541b62501e4" containerID="eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d" exitCode=6
Jul 16 22:44:49.182783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.182651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerDied","Data":"eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d"}
Jul 16 22:44:49.182783 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.182713    2581 scope.go:117] "RemoveContainer" containerID="96fe74c4492b420faf5b8528fbb5b375c1b029f15c94dc34af637cf78c6455f8"
Jul 16 22:44:49.182904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.182868    2581 scope.go:117] "RemoveContainer" containerID="eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d"
Jul 16 22:44:49.183145 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:49.183115    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w_odh-ai-gateway-infra(3f3b309f-2161-4a53-b674-3541b62501e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" podUID="3f3b309f-2161-4a53-b674-3541b62501e4"
Jul 16 22:44:49.184317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.184293    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b72a648-7328-434f-aabe-b5cc626bac96" containerID="5178a1055d40b9f65e63f6df5544e581d81c609339cf555e291eb1ad862c5490" exitCode=6
Jul 16 22:44:49.184400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.184333    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" event={"ID":"9b72a648-7328-434f-aabe-b5cc626bac96","Type":"ContainerDied","Data":"5178a1055d40b9f65e63f6df5544e581d81c609339cf555e291eb1ad862c5490"}
Jul 16 22:44:49.184400 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.184376    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" event={"ID":"9b72a648-7328-434f-aabe-b5cc626bac96","Type":"ContainerStarted","Data":"c889e7556be99678f3ea546b396f8bd91f4b165f5f5042dabdc5c59c07f63936"}
Jul 16 22:44:49.184548 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:49.184535    2581 scope.go:117] "RemoveContainer" containerID="5178a1055d40b9f65e63f6df5544e581d81c609339cf555e291eb1ad862c5490"
Jul 16 22:44:50.191670 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:50.191639    2581 scope.go:117] "RemoveContainer" containerID="eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d"
Jul 16 22:44:50.192135 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:50.191835    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w_odh-ai-gateway-infra(3f3b309f-2161-4a53-b674-3541b62501e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" podUID="3f3b309f-2161-4a53-b674-3541b62501e4"
Jul 16 22:44:50.193122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:50.193095    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b72a648-7328-434f-aabe-b5cc626bac96" containerID="d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71" exitCode=6
Jul 16 22:44:50.193195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:50.193172    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" event={"ID":"9b72a648-7328-434f-aabe-b5cc626bac96","Type":"ContainerDied","Data":"d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71"}
Jul 16 22:44:50.193258 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:50.193246    2581 scope.go:117] "RemoveContainer" containerID="5178a1055d40b9f65e63f6df5544e581d81c609339cf555e291eb1ad862c5490"
Jul 16 22:44:50.193346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:50.193333    2581 scope.go:117] "RemoveContainer" containerID="d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71"
Jul 16 22:44:50.193500 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:50.193476    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5_odh-ai-gateway-infra(9b72a648-7328-434f-aabe-b5cc626bac96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" podUID="9b72a648-7328-434f-aabe-b5cc626bac96"
Jul 16 22:44:51.199187 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.199156    2581 scope.go:117] "RemoveContainer" containerID="d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71"
Jul 16 22:44:51.199655 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:51.199383    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5_odh-ai-gateway-infra(9b72a648-7328-434f-aabe-b5cc626bac96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" podUID="9b72a648-7328-434f-aabe-b5cc626bac96"
Jul 16 22:44:51.225808 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.225768    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" podUID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" containerName="main" containerID="cri-o://4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64" gracePeriod=45
Jul 16 22:44:51.478815 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.478788    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:44:51.501825 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.501799    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.501840    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.501911    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bhxmr\" (UniqueName: \"kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.501999    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502165 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502026    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502165 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502051    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm\") pod \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\" (UID: \"fa949b46-fa04-4418-8ae8-3dbe083b83e5\") "
Jul 16 22:44:51.502165 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502147    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home" (OuterVolumeSpecName: "home") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:51.502376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502356    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:51.502502 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502472    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:51.502574 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.502541    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache" (OuterVolumeSpecName: "model-cache") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:51.504332 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.504308    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:44:51.504432 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.504341    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr" (OuterVolumeSpecName: "kube-api-access-bhxmr") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "kube-api-access-bhxmr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:44:51.504582 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.504565    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm" (OuterVolumeSpecName: "dshm") pod "fa949b46-fa04-4418-8ae8-3dbe083b83e5" (UID: "fa949b46-fa04-4418-8ae8-3dbe083b83e5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:44:51.603031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.602993    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:51.603031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.603025    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:51.603031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.603033    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/fa949b46-fa04-4418-8ae8-3dbe083b83e5-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:51.603031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.603040    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/fa949b46-fa04-4418-8ae8-3dbe083b83e5-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:51.603363 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:51.603049    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bhxmr\" (UniqueName: \"kubernetes.io/projected/fa949b46-fa04-4418-8ae8-3dbe083b83e5-kube-api-access-bhxmr\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:44:52.206004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.205973    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" containerID="4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64" exitCode=0
Jul 16 22:44:52.206593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.206014    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"
Jul 16 22:44:52.206593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.206033    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" event={"ID":"fa949b46-fa04-4418-8ae8-3dbe083b83e5","Type":"ContainerDied","Data":"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"}
Jul 16 22:44:52.206593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.206071    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j" event={"ID":"fa949b46-fa04-4418-8ae8-3dbe083b83e5","Type":"ContainerDied","Data":"b529bf7bce73c7acbd71ac3a3e0dc25d5df623971c0536be56f4b70483971c7c"}
Jul 16 22:44:52.206593 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.206092    2581 scope.go:117] "RemoveContainer" containerID="4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"
Jul 16 22:44:52.214980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.214959    2581 scope.go:117] "RemoveContainer" containerID="4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"
Jul 16 22:44:52.215240 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:44:52.215213    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64\": container with ID starting with 4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64 not found: ID does not exist" containerID="4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"
Jul 16 22:44:52.215291 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.215250    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64"} err="failed to get container status \"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64\": rpc error: code = NotFound desc = could not find container \"4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64\": container with ID starting with 4aa7ae325297533ea8f5e9084a14fac97705308c349d3e5e839a5b0703b14e64 not found: ID does not exist"
Jul 16 22:44:52.230393 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.230367    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:44:52.232326 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:52.232299    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-fbed9b/auth-test-model-fbed9b-kserve-666544ccc9-hvd5j"]
Jul 16 22:44:53.159874 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:53.159838    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" path="/var/lib/kubelet/pods/fa949b46-fa04-4418-8ae8-3dbe083b83e5/volumes"
Jul 16 22:44:57.803763 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:44:57.803728    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:45:00.191330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.191292    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"]
Jul 16 22:45:00.191726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.191660    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" containerName="main"
Jul 16 22:45:00.191726 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.191674    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" containerName="main"
Jul 16 22:45:00.191803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.191734    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa949b46-fa04-4418-8ae8-3dbe083b83e5" containerName="main"
Jul 16 22:45:00.196462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.196444    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"]
Jul 16 22:45:00.196581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.196561    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"
Jul 16 22:45:00.205426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.205395    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"
Jul 16 22:45:00.211180 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.211156    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"
Jul 16 22:45:00.217193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.217175    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"
Jul 16 22:45:00.242419 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.241818    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"]
Jul 16 22:45:00.243080 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.243020    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"]
Jul 16 22:45:00.377495 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.377433    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"]
Jul 16 22:45:00.379888 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:00.379853    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod317006b8_7a96_4592_b2a8_48fba5c7d1d0.slice/crio-cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666 WatchSource:0}: Error finding container cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666: Status 404 returned error can't find the container with id cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666
Jul 16 22:45:00.414662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:00.414637    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"]
Jul 16 22:45:00.418418 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:00.418393    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95a134b0_ecab_47be_8243_1cbe1947933f.slice/crio-866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6 WatchSource:0}: Error finding container 866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6: Status 404 returned error can't find the container with id 866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6
Jul 16 22:45:01.258550 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.258500    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd" event={"ID":"317006b8-7a96-4592-b2a8-48fba5c7d1d0","Type":"ContainerStarted","Data":"401412b0d1e59b128a473aacce964b65ff9ad99a18107b707392962d45dea6e0"}
Jul 16 22:45:01.259016 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.258562    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd" event={"ID":"317006b8-7a96-4592-b2a8-48fba5c7d1d0","Type":"ContainerStarted","Data":"cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666"}
Jul 16 22:45:01.260105 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.260078    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7" event={"ID":"95a134b0-ecab-47be-8243-1cbe1947933f","Type":"ContainerStarted","Data":"4cf970172fa865c60aa831c13d19a2976351a98188bdcea9052802bb2463e8df"}
Jul 16 22:45:01.260250 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.260128    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7" event={"ID":"95a134b0-ecab-47be-8243-1cbe1947933f","Type":"ContainerStarted","Data":"866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6"}
Jul 16 22:45:01.310373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.310321    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7" podStartSLOduration=1.310304842 podStartE2EDuration="1.310304842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:01.310215675 +0000 UTC m=+3488.639655578" watchObservedRunningTime="2026-07-16 22:45:01.310304842 +0000 UTC m=+3488.639744745"
Jul 16 22:45:01.311368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:01.311338    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd" podStartSLOduration=1.311322852 podStartE2EDuration="1.311322852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:01.283827586 +0000 UTC m=+3488.613267490" watchObservedRunningTime="2026-07-16 22:45:01.311322852 +0000 UTC m=+3488.640762754"
Jul 16 22:45:02.266354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:02.266318    2581 generic.go:356] "Generic (PLEG): container finished" podID="95a134b0-ecab-47be-8243-1cbe1947933f" containerID="4cf970172fa865c60aa831c13d19a2976351a98188bdcea9052802bb2463e8df" exitCode=0
Jul 16 22:45:02.266769 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:02.266399    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7" event={"ID":"95a134b0-ecab-47be-8243-1cbe1947933f","Type":"ContainerDied","Data":"4cf970172fa865c60aa831c13d19a2976351a98188bdcea9052802bb2463e8df"}
Jul 16 22:45:02.267795 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:02.267773    2581 generic.go:356] "Generic (PLEG): container finished" podID="317006b8-7a96-4592-b2a8-48fba5c7d1d0" containerID="401412b0d1e59b128a473aacce964b65ff9ad99a18107b707392962d45dea6e0" exitCode=0
Jul 16 22:45:02.267904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:02.267803    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd" event={"ID":"317006b8-7a96-4592-b2a8-48fba5c7d1d0","Type":"ContainerDied","Data":"401412b0d1e59b128a473aacce964b65ff9ad99a18107b707392962d45dea6e0"}
Jul 16 22:45:03.157399 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.157361    2581 scope.go:117] "RemoveContainer" containerID="d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71"
Jul 16 22:45:03.434622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.434599    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"
Jul 16 22:45:03.436477 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.436462    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"
Jul 16 22:45:03.769742 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.769662    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:03.770015 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770002    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95a134b0-ecab-47be-8243-1cbe1947933f" containerName="cleanup"
Jul 16 22:45:03.770083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770016    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="95a134b0-ecab-47be-8243-1cbe1947933f" containerName="cleanup"
Jul 16 22:45:03.770083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770027    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="317006b8-7a96-4592-b2a8-48fba5c7d1d0" containerName="cleanup"
Jul 16 22:45:03.770083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770033    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="317006b8-7a96-4592-b2a8-48fba5c7d1d0" containerName="cleanup"
Jul 16 22:45:03.770267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770089    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="317006b8-7a96-4592-b2a8-48fba5c7d1d0" containerName="cleanup"
Jul 16 22:45:03.770267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.770097    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="95a134b0-ecab-47be-8243-1cbe1947933f" containerName="cleanup"
Jul 16 22:45:03.773322 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.773303    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:03.781781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.781749    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:03.807288 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.807256    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zkhh6\" (UID: \"165d2e07-8696-4196-a42f-ede743609f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:03.908617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.908581    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zkhh6\" (UID: \"165d2e07-8696-4196-a42f-ede743609f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:03.909178 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:03.909159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-zkhh6\" (UID: \"165d2e07-8696-4196-a42f-ede743609f6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:04.085410 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.085326    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:04.218715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.218678    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:04.220310 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:04.220280    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165d2e07_8696_4196_a42f_ede743609f6d.slice/crio-1142c59dd41d27bd5fe4f3d679e5627c9070fb2e902bfcb4db8f2f228136ba62 WatchSource:0}: Error finding container 1142c59dd41d27bd5fe4f3d679e5627c9070fb2e902bfcb4db8f2f228136ba62: Status 404 returned error can't find the container with id 1142c59dd41d27bd5fe4f3d679e5627c9070fb2e902bfcb4db8f2f228136ba62
Jul 16 22:45:04.277893 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.277847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" event={"ID":"165d2e07-8696-4196-a42f-ede743609f6d","Type":"ContainerStarted","Data":"1142c59dd41d27bd5fe4f3d679e5627c9070fb2e902bfcb4db8f2f228136ba62"}
Jul 16 22:45:04.279580 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.279550    2581 generic.go:356] "Generic (PLEG): container finished" podID="9b72a648-7328-434f-aabe-b5cc626bac96" containerID="5022abee206637409ae1b6e46754ce702d3c2a9456b3c82df3180c065b6bb28c" exitCode=6
Jul 16 22:45:04.279694 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.279630    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" event={"ID":"9b72a648-7328-434f-aabe-b5cc626bac96","Type":"ContainerDied","Data":"5022abee206637409ae1b6e46754ce702d3c2a9456b3c82df3180c065b6bb28c"}
Jul 16 22:45:04.279694 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.279690    2581 scope.go:117] "RemoveContainer" containerID="d0b236826902d542bae08c79e827b30544625fa3a6bf45f95b7c01bcf91e1e71"
Jul 16 22:45:04.279967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.279944    2581 scope.go:117] "RemoveContainer" containerID="5022abee206637409ae1b6e46754ce702d3c2a9456b3c82df3180c065b6bb28c"
Jul 16 22:45:04.280185 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:04.280163    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5_odh-ai-gateway-infra(9b72a648-7328-434f-aabe-b5cc626bac96)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" podUID="9b72a648-7328-434f-aabe-b5cc626bac96"
Jul 16 22:45:04.280913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.280897    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"
Jul 16 22:45:04.281009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.280936    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7" event={"ID":"95a134b0-ecab-47be-8243-1cbe1947933f","Type":"ContainerDied","Data":"866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6"}
Jul 16 22:45:04.281009 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.280976    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="866dc7e4ec5da278f3b363fe99333667698c6d99240930a2c6c1b9385ef26eb6"
Jul 16 22:45:04.282143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.282115    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd" event={"ID":"317006b8-7a96-4592-b2a8-48fba5c7d1d0","Type":"ContainerDied","Data":"cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666"}
Jul 16 22:45:04.282143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.282136    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"
Jul 16 22:45:04.282321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:04.282150    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd91181ba6255bb1b7ad79f2dc648ec803297fd7d073dfa287adb0bc8d48c666"
Jul 16 22:45:05.288479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.288377    2581 generic.go:356] "Generic (PLEG): container finished" podID="165d2e07-8696-4196-a42f-ede743609f6d" containerID="fc3e042083b08d4219a7433837cdf8d87d24e1413d9052f5392667e66d4cabf7" exitCode=6
Jul 16 22:45:05.288479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.288464    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" event={"ID":"165d2e07-8696-4196-a42f-ede743609f6d","Type":"ContainerDied","Data":"fc3e042083b08d4219a7433837cdf8d87d24e1413d9052f5392667e66d4cabf7"}
Jul 16 22:45:05.289008 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.288648    2581 scope.go:117] "RemoveContainer" containerID="fc3e042083b08d4219a7433837cdf8d87d24e1413d9052f5392667e66d4cabf7"
Jul 16 22:45:05.310907 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.310881    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:45:05.446372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.446344    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:45:05.522912 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.522879    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca\") pod \"9b72a648-7328-434f-aabe-b5cc626bac96\" (UID: \"9b72a648-7328-434f-aabe-b5cc626bac96\") "
Jul 16 22:45:05.523266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.523214    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9b72a648-7328-434f-aabe-b5cc626bac96" (UID: "9b72a648-7328-434f-aabe-b5cc626bac96"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:05.624199 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:05.624161    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9b72a648-7328-434f-aabe-b5cc626bac96-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:45:06.154748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.154720    2581 scope.go:117] "RemoveContainer" containerID="eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d"
Jul 16 22:45:06.297666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.297566    2581 generic.go:356] "Generic (PLEG): container finished" podID="165d2e07-8696-4196-a42f-ede743609f6d" containerID="0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87" exitCode=6
Jul 16 22:45:06.297666 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.297651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" event={"ID":"165d2e07-8696-4196-a42f-ede743609f6d","Type":"ContainerDied","Data":"0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87"}
Jul 16 22:45:06.298181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.297718    2581 scope.go:117] "RemoveContainer" containerID="fc3e042083b08d4219a7433837cdf8d87d24e1413d9052f5392667e66d4cabf7"
Jul 16 22:45:06.298181 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.297936    2581 scope.go:117] "RemoveContainer" containerID="0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87"
Jul 16 22:45:06.298321 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:06.298188    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zkhh6_odh-ai-gateway-infra(165d2e07-8696-4196-a42f-ede743609f6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" podUID="165d2e07-8696-4196-a42f-ede743609f6d"
Jul 16 22:45:06.298978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.298935    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"
Jul 16 22:45:06.298978 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.298947    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5" event={"ID":"9b72a648-7328-434f-aabe-b5cc626bac96","Type":"ContainerDied","Data":"c889e7556be99678f3ea546b396f8bd91f4b165f5f5042dabdc5c59c07f63936"}
Jul 16 22:45:06.309917 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.309900    2581 scope.go:117] "RemoveContainer" containerID="5022abee206637409ae1b6e46754ce702d3c2a9456b3c82df3180c065b6bb28c"
Jul 16 22:45:06.333761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.333724    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:45:06.342485 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:06.342460    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-5t8c5"]
Jul 16 22:45:07.160094 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.160057    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" path="/var/lib/kubelet/pods/9b72a648-7328-434f-aabe-b5cc626bac96/volumes"
Jul 16 22:45:07.310197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.310171    2581 scope.go:117] "RemoveContainer" containerID="0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87"
Jul 16 22:45:07.310647 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:07.310404    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zkhh6_odh-ai-gateway-infra(165d2e07-8696-4196-a42f-ede743609f6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" podUID="165d2e07-8696-4196-a42f-ede743609f6d"
Jul 16 22:45:07.311635 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.311609    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f3b309f-2161-4a53-b674-3541b62501e4" containerID="a9ecdd084da3c42c39ae127ed6f9620884aebc2b7b685a5df405e7b411f3754e" exitCode=6
Jul 16 22:45:07.311758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.311679    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerDied","Data":"a9ecdd084da3c42c39ae127ed6f9620884aebc2b7b685a5df405e7b411f3754e"}
Jul 16 22:45:07.311758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.311732    2581 scope.go:117] "RemoveContainer" containerID="eef608d50fa90675b2cc8452fff7f05cd2065228184378ab18d2eaaf54837d8d"
Jul 16 22:45:07.312034 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:07.312018    2581 scope.go:117] "RemoveContainer" containerID="a9ecdd084da3c42c39ae127ed6f9620884aebc2b7b685a5df405e7b411f3754e"
Jul 16 22:45:07.312258 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:07.312210    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w_odh-ai-gateway-infra(3f3b309f-2161-4a53-b674-3541b62501e4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" podUID="3f3b309f-2161-4a53-b674-3541b62501e4"
Jul 16 22:45:08.352195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:08.352162    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:45:08.480300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:08.480274    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:45:08.550490 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:08.550452    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca\") pod \"3f3b309f-2161-4a53-b674-3541b62501e4\" (UID: \"3f3b309f-2161-4a53-b674-3541b62501e4\") "
Jul 16 22:45:08.550826 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:08.550792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f3b309f-2161-4a53-b674-3541b62501e4" (UID: "3f3b309f-2161-4a53-b674-3541b62501e4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:08.651879 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:08.651783    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f3b309f-2161-4a53-b674-3541b62501e4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:45:09.324811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:09.324782    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"
Jul 16 22:45:09.324995 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:09.324810    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w" event={"ID":"3f3b309f-2161-4a53-b674-3541b62501e4","Type":"ContainerDied","Data":"9569ebe0f794f8c0b6a06372db68157db27b1339d74e3490ad356fb2aa51c50e"}
Jul 16 22:45:09.324995 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:09.324875    2581 scope.go:117] "RemoveContainer" containerID="a9ecdd084da3c42c39ae127ed6f9620884aebc2b7b685a5df405e7b411f3754e"
Jul 16 22:45:09.343715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:09.343687    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:45:09.349835 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:09.349814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-99l9w"]
Jul 16 22:45:11.159724 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:11.159694    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" path="/var/lib/kubelet/pods/3f3b309f-2161-4a53-b674-3541b62501e4/volumes"
Jul 16 22:45:13.207929 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.207898    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208238    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208252    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208263    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208269    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208278    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208284    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208293    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208298    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208303    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208309    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208322    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208327    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208378    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.208737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208386    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208395    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208401    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f3b309f-2161-4a53-b674-3541b62501e4" containerName="revoke-keys"
Jul 16 22:45:13.208737 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.208407    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:13.212814 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.212796    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.215749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.215727    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"kube-root-ca.crt\""
Jul 16 22:45:13.216479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.216458    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"openshift-service-ca.crt\""
Jul 16 22:45:13.216709 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.216689    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"default-dockercfg-25fwx\""
Jul 16 22:45:13.216796 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.216733    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-106c85\"/\"sub-test-model-106c85-kserve-self-signed-certs\""
Jul 16 22:45:13.225900 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.225875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:45:13.294066 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294030    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.294321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294080    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.294321 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294151    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.294468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294341    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.294468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294381    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlvq7\" (UniqueName: \"kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.294468 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.294410    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395542    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395602    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395651    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395690    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395714    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlvq7\" (UniqueName: \"kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.395801 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.395744    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.396077 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.396028    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.396138 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.396104    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.396204 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.396136    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.398091 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.398069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.398450 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.398429    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.405999 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.405966    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlvq7\" (UniqueName: \"kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7\") pod \"sub-test-model-106c85-kserve-549f9fd5d5-t2vmz\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") " pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.522376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.522278    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:13.686295 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:13.686264    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:45:13.687107 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:13.687074    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd78078aa_6935_41b1_8229_b00b2189394e.slice/crio-13b568ec8b5416bb8163022c4591a50afad23c3b1f05fbfabf4520fbb9e4f4a0 WatchSource:0}: Error finding container 13b568ec8b5416bb8163022c4591a50afad23c3b1f05fbfabf4520fbb9e4f4a0: Status 404 returned error can't find the container with id 13b568ec8b5416bb8163022c4591a50afad23c3b1f05fbfabf4520fbb9e4f4a0
Jul 16 22:45:14.353363 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:14.353263    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" event={"ID":"d78078aa-6935-41b1-8229-b00b2189394e","Type":"ContainerStarted","Data":"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"}
Jul 16 22:45:14.353363 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:14.353321    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" event={"ID":"d78078aa-6935-41b1-8229-b00b2189394e","Type":"ContainerStarted","Data":"13b568ec8b5416bb8163022c4591a50afad23c3b1f05fbfabf4520fbb9e4f4a0"}
Jul 16 22:45:14.376502 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:14.376452    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" podStartSLOduration=1.376436423 podStartE2EDuration="1.376436423s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:14.374850865 +0000 UTC m=+3501.704290789" watchObservedRunningTime="2026-07-16 22:45:14.376436423 +0000 UTC m=+3501.705876345"
Jul 16 22:45:17.917369 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:17.917332    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:45:19.155365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:19.155324    2581 scope.go:117] "RemoveContainer" containerID="0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87"
Jul 16 22:45:20.384462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:20.384424    2581 generic.go:356] "Generic (PLEG): container finished" podID="165d2e07-8696-4196-a42f-ede743609f6d" containerID="e89a1d282b81c32c05f2712b9282d5d2b26442c980c2147a698e5cf9b3946c7a" exitCode=6
Jul 16 22:45:20.384934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:20.384461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" event={"ID":"165d2e07-8696-4196-a42f-ede743609f6d","Type":"ContainerDied","Data":"e89a1d282b81c32c05f2712b9282d5d2b26442c980c2147a698e5cf9b3946c7a"}
Jul 16 22:45:20.384934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:20.384524    2581 scope.go:117] "RemoveContainer" containerID="0e3b96bc6e5735c0a2e22a8238a639abbeca04300f54b0f2a2fe8ce86e98fb87"
Jul 16 22:45:20.384934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:20.384819    2581 scope.go:117] "RemoveContainer" containerID="e89a1d282b81c32c05f2712b9282d5d2b26442c980c2147a698e5cf9b3946c7a"
Jul 16 22:45:20.385104 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:20.385033    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-zkhh6_odh-ai-gateway-infra(165d2e07-8696-4196-a42f-ede743609f6d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" podUID="165d2e07-8696-4196-a42f-ede743609f6d"
Jul 16 22:45:21.418662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:21.418627    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:21.551041 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:21.551007    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:21.668301 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:21.668272    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca\") pod \"165d2e07-8696-4196-a42f-ede743609f6d\" (UID: \"165d2e07-8696-4196-a42f-ede743609f6d\") "
Jul 16 22:45:21.668632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:21.668608    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "165d2e07-8696-4196-a42f-ede743609f6d" (UID: "165d2e07-8696-4196-a42f-ede743609f6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:21.769277 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:21.769179    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/165d2e07-8696-4196-a42f-ede743609f6d-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:45:22.397818 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:22.397782    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"
Jul 16 22:45:22.397818 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:22.397780    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6" event={"ID":"165d2e07-8696-4196-a42f-ede743609f6d","Type":"ContainerDied","Data":"1142c59dd41d27bd5fe4f3d679e5627c9070fb2e902bfcb4db8f2f228136ba62"}
Jul 16 22:45:22.398076 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:22.397852    2581 scope.go:117] "RemoveContainer" containerID="e89a1d282b81c32c05f2712b9282d5d2b26442c980c2147a698e5cf9b3946c7a"
Jul 16 22:45:22.421637 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:22.421605    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:22.423809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:22.423789    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-zkhh6"]
Jul 16 22:45:23.160838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:23.160804    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="165d2e07-8696-4196-a42f-ede743609f6d" path="/var/lib/kubelet/pods/165d2e07-8696-4196-a42f-ede743609f6d/volumes"
Jul 16 22:45:23.522823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:23.522736    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:23.522823 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:23.522782    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:23.535389 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:23.535366    2581 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:24.419838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:24.419805    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:45:25.443590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443559    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443872    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443881    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443893    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443898    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443907    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.443948 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443912    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.444195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443968    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.444195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443978    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:25.444195 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.443984    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b72a648-7328-434f-aabe-b5cc626bac96" containerName="revoke-keys"
Jul 16 22:45:25.448437 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.448420    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:25.450935 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.450917    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:45:25.451055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.450941    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:45:25.455943 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.455917    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:25.604296 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.604257    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm\" (UID: \"5eb49047-b75f-451d-bc88-f6d966916c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:25.705585 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.705499    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm\" (UID: \"5eb49047-b75f-451d-bc88-f6d966916c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:25.706110 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.706092    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm\" (UID: \"5eb49047-b75f-451d-bc88-f6d966916c1e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:25.761010 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.760984    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:25.890608 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:25.890578    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:25.892376 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:25.892350    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5eb49047_b75f_451d_bc88_f6d966916c1e.slice/crio-2028a059726141ddf97b214b59111e7415f6ec7b57fb686cd86feb0f9e00ab06 WatchSource:0}: Error finding container 2028a059726141ddf97b214b59111e7415f6ec7b57fb686cd86feb0f9e00ab06: Status 404 returned error can't find the container with id 2028a059726141ddf97b214b59111e7415f6ec7b57fb686cd86feb0f9e00ab06
Jul 16 22:45:26.419111 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:26.419074    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerStarted","Data":"76506e04ac77580ca6710a21e64a81d1fe0a7895fc36204ee40f93144ee4e44d"}
Jul 16 22:45:26.419328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:26.419130    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerStarted","Data":"2028a059726141ddf97b214b59111e7415f6ec7b57fb686cd86feb0f9e00ab06"}
Jul 16 22:45:26.436280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:26.436208    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" podStartSLOduration=1.436192563 podStartE2EDuration="1.436192563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:26.434647259 +0000 UTC m=+3513.764087164" watchObservedRunningTime="2026-07-16 22:45:26.436192563 +0000 UTC m=+3513.765632465"
Jul 16 22:45:27.425791 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:27.425751    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerID="76506e04ac77580ca6710a21e64a81d1fe0a7895fc36204ee40f93144ee4e44d" exitCode=6
Jul 16 22:45:27.426266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:27.425844    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerDied","Data":"76506e04ac77580ca6710a21e64a81d1fe0a7895fc36204ee40f93144ee4e44d"}
Jul 16 22:45:27.426266 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:27.425978    2581 scope.go:117] "RemoveContainer" containerID="76506e04ac77580ca6710a21e64a81d1fe0a7895fc36204ee40f93144ee4e44d"
Jul 16 22:45:28.432591 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:28.432553    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerID="ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070" exitCode=6
Jul 16 22:45:28.433023 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:28.432645    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerDied","Data":"ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070"}
Jul 16 22:45:28.433023 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:28.432704    2581 scope.go:117] "RemoveContainer" containerID="76506e04ac77580ca6710a21e64a81d1fe0a7895fc36204ee40f93144ee4e44d"
Jul 16 22:45:28.433023 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:28.432827    2581 scope.go:117] "RemoveContainer" containerID="ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070"
Jul 16 22:45:28.433140 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:28.433053    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm_odh-ai-gateway-infra(5eb49047-b75f-451d-bc88-f6d966916c1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e"
Jul 16 22:45:29.439106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:29.439075    2581 scope.go:117] "RemoveContainer" containerID="ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070"
Jul 16 22:45:29.439551 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:29.439297    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm_odh-ai-gateway-infra(5eb49047-b75f-451d-bc88-f6d966916c1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e"
Jul 16 22:45:41.921843 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:41.921807    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:45:43.158372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:43.158337    2581 scope.go:117] "RemoveContainer" containerID="ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070"
Jul 16 22:45:43.508760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:43.508675    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerStarted","Data":"0c1932535ee826648d4e7c7b946a3020da569369707e504cad22fec8ba24c310"}
Jul 16 22:45:44.515871 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:44.515841    2581 generic.go:356] "Generic (PLEG): container finished" podID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerID="0c1932535ee826648d4e7c7b946a3020da569369707e504cad22fec8ba24c310" exitCode=6
Jul 16 22:45:44.516356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:44.515920    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerDied","Data":"0c1932535ee826648d4e7c7b946a3020da569369707e504cad22fec8ba24c310"}
Jul 16 22:45:44.516356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:44.516026    2581 scope.go:117] "RemoveContainer" containerID="ecd04f59754d7d8c0a90b4e7b22a48799b86bf3b37f6dfb9e28ff2aa7572d070"
Jul 16 22:45:44.516509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:44.516403    2581 scope.go:117] "RemoveContainer" containerID="0c1932535ee826648d4e7c7b946a3020da569369707e504cad22fec8ba24c310"
Jul 16 22:45:44.516633 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:44.516614    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm_odh-ai-gateway-infra(5eb49047-b75f-451d-bc88-f6d966916c1e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e"
Jul 16 22:45:44.542758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:44.542735    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:45.662221 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:45.662189    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:45.682405 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:45.682378    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca\") pod \"5eb49047-b75f-451d-bc88-f6d966916c1e\" (UID: \"5eb49047-b75f-451d-bc88-f6d966916c1e\") "
Jul 16 22:45:45.682757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:45.682725    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5eb49047-b75f-451d-bc88-f6d966916c1e" (UID: "5eb49047-b75f-451d-bc88-f6d966916c1e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:45.783989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:45.783908    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5eb49047-b75f-451d-bc88-f6d966916c1e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:45:46.528841 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:46.528809    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"
Jul 16 22:45:46.528997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:46.528846    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm" event={"ID":"5eb49047-b75f-451d-bc88-f6d966916c1e","Type":"ContainerDied","Data":"2028a059726141ddf97b214b59111e7415f6ec7b57fb686cd86feb0f9e00ab06"}
Jul 16 22:45:46.528997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:46.528934    2581 scope.go:117] "RemoveContainer" containerID="0c1932535ee826648d4e7c7b946a3020da569369707e504cad22fec8ba24c310"
Jul 16 22:45:46.551324 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:46.551291    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:46.553168 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:46.553141    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-xdqbm"]
Jul 16 22:45:47.160729 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:47.160691    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" path="/var/lib/kubelet/pods/5eb49047-b75f-451d-bc88-f6d966916c1e/volumes"
Jul 16 22:45:53.923744 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.923702    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:45:53.924223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924178    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924197    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924251    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924223 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924260    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924340    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924353    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="165d2e07-8696-4196-a42f-ede743609f6d" containerName="revoke-keys"
Jul 16 22:45:53.924497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924364    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.924497 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.924372    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:45:53.929346 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.929324    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:45:53.933817 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.933793    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:45:53.946297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.946276    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:45:53.961904 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:53.961875    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:45:54.061317 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:54.061198    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-8kw6d\" (UID: \"aff542f7-4edc-4ec4-9242-7f2d9b635dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:45:54.162734 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:54.162697    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-8kw6d\" (UID: \"aff542f7-4edc-4ec4-9242-7f2d9b635dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:45:54.163380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:54.163359    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-8kw6d\" (UID: \"aff542f7-4edc-4ec4-9242-7f2d9b635dcb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:45:54.241017 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:54.240985    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:45:54.575915 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:54.575885    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:45:54.577194 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:45:54.577167    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaff542f7_4edc_4ec4_9242_7f2d9b635dcb.slice/crio-ea627289c2d186a6cd15d154e617cbf6a0f53f4f1ef76f0b02c759770eefa4e1 WatchSource:0}: Error finding container ea627289c2d186a6cd15d154e617cbf6a0f53f4f1ef76f0b02c759770eefa4e1: Status 404 returned error can't find the container with id ea627289c2d186a6cd15d154e617cbf6a0f53f4f1ef76f0b02c759770eefa4e1
Jul 16 22:45:55.575545 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:55.575455    2581 generic.go:356] "Generic (PLEG): container finished" podID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerID="c8cd1930adc1fb5179df148b055be161668c800e1356a7796eebc93e2d3eed3a" exitCode=6
Jul 16 22:45:55.575909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:55.575549    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerDied","Data":"c8cd1930adc1fb5179df148b055be161668c800e1356a7796eebc93e2d3eed3a"}
Jul 16 22:45:55.575909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:55.575600    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerStarted","Data":"ea627289c2d186a6cd15d154e617cbf6a0f53f4f1ef76f0b02c759770eefa4e1"}
Jul 16 22:45:55.575909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:55.575696    2581 scope.go:117] "RemoveContainer" containerID="c8cd1930adc1fb5179df148b055be161668c800e1356a7796eebc93e2d3eed3a"
Jul 16 22:45:56.582122 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:56.582025    2581 generic.go:356] "Generic (PLEG): container finished" podID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerID="e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c" exitCode=6
Jul 16 22:45:56.582544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:56.582117    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerDied","Data":"e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c"}
Jul 16 22:45:56.582544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:56.582175    2581 scope.go:117] "RemoveContainer" containerID="c8cd1930adc1fb5179df148b055be161668c800e1356a7796eebc93e2d3eed3a"
Jul 16 22:45:56.582544 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:56.582421    2581 scope.go:117] "RemoveContainer" containerID="e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c"
Jul 16 22:45:56.582676 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:56.582638    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-8kw6d_odh-ai-gateway-infra(aff542f7-4edc-4ec4-9242-7f2d9b635dcb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb"
Jul 16 22:45:57.595057 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:45:57.595033    2581 scope.go:117] "RemoveContainer" containerID="e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c"
Jul 16 22:45:57.595492 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:45:57.595206    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-8kw6d_odh-ai-gateway-infra(aff542f7-4edc-4ec4-9242-7f2d9b635dcb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb"
Jul 16 22:46:00.528748 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:00.528713    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:46:07.298831 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:07.298797    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:46:09.155381 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:09.155350    2581 scope.go:117] "RemoveContainer" containerID="e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c"
Jul 16 22:46:09.652541 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:09.652506    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerStarted","Data":"e802fecbd3293d2b91026748b2d864f95ff72f009b547c19647cac1b8bade9fd"}
Jul 16 22:46:09.669768 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:09.669719    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" podStartSLOduration=16.669699003 podStartE2EDuration="16.669699003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:09.667442981 +0000 UTC m=+3556.996882883" watchObservedRunningTime="2026-07-16 22:46:09.669699003 +0000 UTC m=+3556.999138905"
Jul 16 22:46:10.658680 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:10.658645    2581 generic.go:356] "Generic (PLEG): container finished" podID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerID="e802fecbd3293d2b91026748b2d864f95ff72f009b547c19647cac1b8bade9fd" exitCode=6
Jul 16 22:46:10.659162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:10.658699    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerDied","Data":"e802fecbd3293d2b91026748b2d864f95ff72f009b547c19647cac1b8bade9fd"}
Jul 16 22:46:10.659162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:10.658776    2581 scope.go:117] "RemoveContainer" containerID="e979ed8bbc29fe4c485919e4fbd917f6bca2de0ab4e0c56d7df52f2186a6082c"
Jul 16 22:46:10.659162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:10.659099    2581 scope.go:117] "RemoveContainer" containerID="e802fecbd3293d2b91026748b2d864f95ff72f009b547c19647cac1b8bade9fd"
Jul 16 22:46:10.659407 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:10.659384    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-8kw6d_odh-ai-gateway-infra(aff542f7-4edc-4ec4-9242-7f2d9b635dcb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb"
Jul 16 22:46:10.686982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:10.686953    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:46:11.798485 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:11.798453    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:46:11.916741 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:11.916670    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca\") pod \"aff542f7-4edc-4ec4-9242-7f2d9b635dcb\" (UID: \"aff542f7-4edc-4ec4-9242-7f2d9b635dcb\") "
Jul 16 22:46:11.916984 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:11.916963    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aff542f7-4edc-4ec4-9242-7f2d9b635dcb" (UID: "aff542f7-4edc-4ec4-9242-7f2d9b635dcb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:12.017793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.017768    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aff542f7-4edc-4ec4-9242-7f2d9b635dcb-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:12.670970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.670934    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d" event={"ID":"aff542f7-4edc-4ec4-9242-7f2d9b635dcb","Type":"ContainerDied","Data":"ea627289c2d186a6cd15d154e617cbf6a0f53f4f1ef76f0b02c759770eefa4e1"}
Jul 16 22:46:12.670970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.670968    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"
Jul 16 22:46:12.671211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.670997    2581 scope.go:117] "RemoveContainer" containerID="e802fecbd3293d2b91026748b2d864f95ff72f009b547c19647cac1b8bade9fd"
Jul 16 22:46:12.692055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.692028    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:46:12.698727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:12.698704    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-8kw6d"]
Jul 16 22:46:13.160817 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:13.160773    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" path="/var/lib/kubelet/pods/aff542f7-4edc-4ec4-9242-7f2d9b635dcb/volumes"
Jul 16 22:46:18.126579 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:18.126543    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:46:22.320309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.320270    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" podUID="d78078aa-6935-41b1-8229-b00b2189394e" containerName="main" containerID="cri-o://4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34" gracePeriod=45
Jul 16 22:46:22.571373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.571309    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:46:22.616658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616634    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.616658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616661    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.616845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616687    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.616845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616740    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.616845 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616769    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.616998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616876    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home" (OuterVolumeSpecName: "home") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:22.616998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.616925    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qlvq7\" (UniqueName: \"kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7\") pod \"d78078aa-6935-41b1-8229-b00b2189394e\" (UID: \"d78078aa-6935-41b1-8229-b00b2189394e\") "
Jul 16 22:46:22.617093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.617016    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache" (OuterVolumeSpecName: "model-cache") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:22.617093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.617026    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:22.617271 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.617222    2581 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-home\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.617376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.617271    2581 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-model-cache\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.617376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.617282    2581 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-tmp-dir\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.618998 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.618978    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm" (OuterVolumeSpecName: "dshm") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:46:22.619110 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.619091    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7" (OuterVolumeSpecName: "kube-api-access-qlvq7") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "kube-api-access-qlvq7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:46:22.619146 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.619093    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d78078aa-6935-41b1-8229-b00b2189394e" (UID: "d78078aa-6935-41b1-8229-b00b2189394e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:46:22.717658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.717630    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlvq7\" (UniqueName: \"kubernetes.io/projected/d78078aa-6935-41b1-8229-b00b2189394e-kube-api-access-qlvq7\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.717658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.717651    2581 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d78078aa-6935-41b1-8229-b00b2189394e-dshm\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.717658 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.717661    2581 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d78078aa-6935-41b1-8229-b00b2189394e-tls-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.720347 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.720322    2581 generic.go:356] "Generic (PLEG): container finished" podID="d78078aa-6935-41b1-8229-b00b2189394e" containerID="4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34" exitCode=0
Jul 16 22:46:22.720467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.720366    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"
Jul 16 22:46:22.720467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.720418    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" event={"ID":"d78078aa-6935-41b1-8229-b00b2189394e","Type":"ContainerDied","Data":"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"}
Jul 16 22:46:22.720467 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.720457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz" event={"ID":"d78078aa-6935-41b1-8229-b00b2189394e","Type":"ContainerDied","Data":"13b568ec8b5416bb8163022c4591a50afad23c3b1f05fbfabf4520fbb9e4f4a0"}
Jul 16 22:46:22.720579 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.720474    2581 scope.go:117] "RemoveContainer" containerID="4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"
Jul 16 22:46:22.729196 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.729176    2581 scope.go:117] "RemoveContainer" containerID="4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"
Jul 16 22:46:22.729452 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:22.729435    2581 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34\": container with ID starting with 4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34 not found: ID does not exist" containerID="4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"
Jul 16 22:46:22.729499 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.729461    2581 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34"} err="failed to get container status \"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34\": rpc error: code = NotFound desc = could not find container \"4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34\": container with ID starting with 4f553774a394e6679949b8c45ee70a189a4955838dd66e79b8b68d4e96aeac34 not found: ID does not exist"
Jul 16 22:46:22.741364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.741333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:46:22.747210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:22.747186    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-106c85/sub-test-model-106c85-kserve-549f9fd5d5-t2vmz"]
Jul 16 22:46:23.160996 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:23.160962    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d78078aa-6935-41b1-8229-b00b2189394e" path="/var/lib/kubelet/pods/d78078aa-6935-41b1-8229-b00b2189394e/volumes"
Jul 16 22:46:25.492337 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492296    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:25.492822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492766    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d78078aa-6935-41b1-8229-b00b2189394e" containerName="main"
Jul 16 22:46:25.492822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492785    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d78078aa-6935-41b1-8229-b00b2189394e" containerName="main"
Jul 16 22:46:25.492822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492805    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.492822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.492822 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492824    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492833    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492847    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492855    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492865    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492874    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="5eb49047-b75f-451d-bc88-f6d966916c1e" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492952    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d78078aa-6935-41b1-8229-b00b2189394e" containerName="main"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492966    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.493100 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.492974    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:25.497522 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.497500    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:25.499945 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.499929    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:46:25.500038 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.499993    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:46:25.504295 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.504269    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:25.538390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.538359    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r\" (UID: \"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:25.639446 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.639418    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r\" (UID: \"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:25.640038 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.640019    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r\" (UID: \"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:25.809298 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:25.809203    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:26.148921 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:26.148890    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:26.152429 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:46:26.152404    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb76bd3b9_1c84_4834_9bc7_0cd27800dfc9.slice/crio-cd97cfe4be6334427083a2f9d1be4183340628dd531e4cffcbf63593336d196b WatchSource:0}: Error finding container cd97cfe4be6334427083a2f9d1be4183340628dd531e4cffcbf63593336d196b: Status 404 returned error can't find the container with id cd97cfe4be6334427083a2f9d1be4183340628dd531e4cffcbf63593336d196b
Jul 16 22:46:26.742838 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:26.742805    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerStarted","Data":"e4025426fe64505824cc465ac278ff335cdcdb2add7fc61d8ee29c0cfcbc6341"}
Jul 16 22:46:26.743297 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:26.742863    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerStarted","Data":"cd97cfe4be6334427083a2f9d1be4183340628dd531e4cffcbf63593336d196b"}
Jul 16 22:46:26.762484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:26.762437    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" podStartSLOduration=1.7624218040000001 podStartE2EDuration="1.762421804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:26.760365144 +0000 UTC m=+3574.089805048" watchObservedRunningTime="2026-07-16 22:46:26.762421804 +0000 UTC m=+3574.091861746"
Jul 16 22:46:27.749372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.749340    2581 generic.go:356] "Generic (PLEG): container finished" podID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerID="e4025426fe64505824cc465ac278ff335cdcdb2add7fc61d8ee29c0cfcbc6341" exitCode=6
Jul 16 22:46:27.749754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.749426    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerDied","Data":"e4025426fe64505824cc465ac278ff335cdcdb2add7fc61d8ee29c0cfcbc6341"}
Jul 16 22:46:27.749754 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.749576    2581 scope.go:117] "RemoveContainer" containerID="e4025426fe64505824cc465ac278ff335cdcdb2add7fc61d8ee29c0cfcbc6341"
Jul 16 22:46:27.930862 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.930829    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:27.931300 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.931274    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="aff542f7-4edc-4ec4-9242-7f2d9b635dcb" containerName="revoke-keys"
Jul 16 22:46:27.935618 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.935598    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:27.943435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.943413    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:27.956492 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:27.956467    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw\" (UID: \"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:28.057376 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.057350    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw\" (UID: \"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:28.057965 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.057946    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw\" (UID: \"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:28.247051 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.247019    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:28.454192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.454166    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:28.456118 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:46:28.456088    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4b1a41f_1f2d_4e3d_ada1_74ca9419b2e8.slice/crio-984cf7bb8c90696a5ca5351d618e6ed6cdeff21505d4a8d56e77c81889ec5ee5 WatchSource:0}: Error finding container 984cf7bb8c90696a5ca5351d618e6ed6cdeff21505d4a8d56e77c81889ec5ee5: Status 404 returned error can't find the container with id 984cf7bb8c90696a5ca5351d618e6ed6cdeff21505d4a8d56e77c81889ec5ee5
Jul 16 22:46:28.756570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.756539    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" event={"ID":"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8","Type":"ContainerStarted","Data":"984cf7bb8c90696a5ca5351d618e6ed6cdeff21505d4a8d56e77c81889ec5ee5"}
Jul 16 22:46:28.758448 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.758419    2581 generic.go:356] "Generic (PLEG): container finished" podID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerID="43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0" exitCode=6
Jul 16 22:46:28.758549 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.758503    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerDied","Data":"43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0"}
Jul 16 22:46:28.758623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.758566    2581 scope.go:117] "RemoveContainer" containerID="e4025426fe64505824cc465ac278ff335cdcdb2add7fc61d8ee29c0cfcbc6341"
Jul 16 22:46:28.758827 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:28.758812    2581 scope.go:117] "RemoveContainer" containerID="43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0"
Jul 16 22:46:28.759073 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:28.759049    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r_odh-ai-gateway-infra(b76bd3b9-1c84-4834-9bc7-0cd27800dfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9"
Jul 16 22:46:29.765093 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:29.765062    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerID="79d174734b3809b54813667b95b8358da293ecf2184264ad05adfff250733933" exitCode=6
Jul 16 22:46:29.765533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:29.765150    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" event={"ID":"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8","Type":"ContainerDied","Data":"79d174734b3809b54813667b95b8358da293ecf2184264ad05adfff250733933"}
Jul 16 22:46:29.765533 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:29.765373    2581 scope.go:117] "RemoveContainer" containerID="79d174734b3809b54813667b95b8358da293ecf2184264ad05adfff250733933"
Jul 16 22:46:29.766989 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:29.766971    2581 scope.go:117] "RemoveContainer" containerID="43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0"
Jul 16 22:46:29.767138 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:29.767121    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r_odh-ai-gateway-infra(b76bd3b9-1c84-4834-9bc7-0cd27800dfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9"
Jul 16 22:46:30.008244 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:30.008196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:46:30.773197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:30.773113    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerID="76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4" exitCode=6
Jul 16 22:46:30.773616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:30.773197    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" event={"ID":"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8","Type":"ContainerDied","Data":"76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4"}
Jul 16 22:46:30.773616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:30.773267    2581 scope.go:117] "RemoveContainer" containerID="79d174734b3809b54813667b95b8358da293ecf2184264ad05adfff250733933"
Jul 16 22:46:30.773616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:30.773507    2581 scope.go:117] "RemoveContainer" containerID="76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4"
Jul 16 22:46:30.773735 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:30.773714    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw_odh-ai-gateway-infra(d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8"
Jul 16 22:46:31.780431 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:31.780404    2581 scope.go:117] "RemoveContainer" containerID="76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4"
Jul 16 22:46:31.780815 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:31.780585    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw_odh-ai-gateway-infra(d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8"
Jul 16 22:46:36.310353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.310316    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:36.314977 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.314959    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:36.325409 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.325385    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:36.434340 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.434311    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf\" (UID: \"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:36.535651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.535622    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf\" (UID: \"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:36.536176 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.536159    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf\" (UID: \"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:36.626308 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.626288    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:36.790214 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:46:36.790184    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode77b07c4_fba4_4ad2_ab8a_a3a86adefcb4.slice/crio-f02273e9014294458defb9366c487cb5c255247f034226e9bdcc31590e0d03d3 WatchSource:0}: Error finding container f02273e9014294458defb9366c487cb5c255247f034226e9bdcc31590e0d03d3: Status 404 returned error can't find the container with id f02273e9014294458defb9366c487cb5c255247f034226e9bdcc31590e0d03d3
Jul 16 22:46:36.793942 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.793914    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:36.809098 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:36.809067    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" event={"ID":"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4","Type":"ContainerStarted","Data":"f02273e9014294458defb9366c487cb5c255247f034226e9bdcc31590e0d03d3"}
Jul 16 22:46:37.815611 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:37.815528    2581 generic.go:356] "Generic (PLEG): container finished" podID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerID="5115f29ae75835065e89ce2e2e8fa868d6d5ac8f1d2ade05c4c82cc98dedef2e" exitCode=6
Jul 16 22:46:37.815983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:37.815615    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" event={"ID":"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4","Type":"ContainerDied","Data":"5115f29ae75835065e89ce2e2e8fa868d6d5ac8f1d2ade05c4c82cc98dedef2e"}
Jul 16 22:46:37.815983 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:37.815778    2581 scope.go:117] "RemoveContainer" containerID="5115f29ae75835065e89ce2e2e8fa868d6d5ac8f1d2ade05c4c82cc98dedef2e"
Jul 16 22:46:38.821740 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:38.821655    2581 generic.go:356] "Generic (PLEG): container finished" podID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerID="6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b" exitCode=6
Jul 16 22:46:38.822142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:38.821745    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" event={"ID":"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4","Type":"ContainerDied","Data":"6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b"}
Jul 16 22:46:38.822142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:38.821808    2581 scope.go:117] "RemoveContainer" containerID="5115f29ae75835065e89ce2e2e8fa868d6d5ac8f1d2ade05c4c82cc98dedef2e"
Jul 16 22:46:38.822142 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:38.822010    2581 scope.go:117] "RemoveContainer" containerID="6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b"
Jul 16 22:46:38.822345 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:38.822253    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf_odh-ai-gateway-infra(e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4"
Jul 16 22:46:39.829124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:39.829095    2581 scope.go:117] "RemoveContainer" containerID="6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b"
Jul 16 22:46:39.829570 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:39.829379    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf_odh-ai-gateway-infra(e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4"
Jul 16 22:46:40.959511 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:40.959475    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:46:42.154894 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:42.154869    2581 scope.go:117] "RemoveContainer" containerID="43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0"
Jul 16 22:46:42.845179 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:42.845144    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerStarted","Data":"7223a270c3e57ec13fa37c4d5dfb836592ad6a73382f93e430971b3a4548237f"}
Jul 16 22:46:43.851440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.851406    2581 generic.go:356] "Generic (PLEG): container finished" podID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerID="7223a270c3e57ec13fa37c4d5dfb836592ad6a73382f93e430971b3a4548237f" exitCode=6
Jul 16 22:46:43.851934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.851470    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerDied","Data":"7223a270c3e57ec13fa37c4d5dfb836592ad6a73382f93e430971b3a4548237f"}
Jul 16 22:46:43.851934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.851518    2581 scope.go:117] "RemoveContainer" containerID="43cff6127724cd446c5cbd000cbe0a6a4af039793b37b2d861d43734887e9fc0"
Jul 16 22:46:43.851934 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.851802    2581 scope.go:117] "RemoveContainer" containerID="7223a270c3e57ec13fa37c4d5dfb836592ad6a73382f93e430971b3a4548237f"
Jul 16 22:46:43.852098 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:43.852016    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r_odh-ai-gateway-infra(b76bd3b9-1c84-4834-9bc7-0cd27800dfc9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9"
Jul 16 22:46:43.880050 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.880023    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:43.895364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.895332    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:46:43.899720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.899704    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:46:43.914002 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.913982    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:46:44.000012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:43.999966    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-cbc4w\" (UID: \"541ae4f9-7440-4d96-ba32-1c70bd270702\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:46:44.101126 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.101100    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-cbc4w\" (UID: \"541ae4f9-7440-4d96-ba32-1c70bd270702\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:46:44.101718 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.101670    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-cbc4w\" (UID: \"541ae4f9-7440-4d96-ba32-1c70bd270702\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:46:44.209644 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.209621    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:46:44.356521 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.356486    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:46:44.357949 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:46:44.357920    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod541ae4f9_7440_4d96_ba32_1c70bd270702.slice/crio-58d40b63b82510a9f4dfceb01f6970ea4da259cda708a9384f6ccc4092a0dd64 WatchSource:0}: Error finding container 58d40b63b82510a9f4dfceb01f6970ea4da259cda708a9384f6ccc4092a0dd64: Status 404 returned error can't find the container with id 58d40b63b82510a9f4dfceb01f6970ea4da259cda708a9384f6ccc4092a0dd64
Jul 16 22:46:44.857616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.857577    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerStarted","Data":"f24c83cb7f9f1f139743779700e5140a5971d212e24d05b557bccdf7ac3988a1"}
Jul 16 22:46:44.858067 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.857647    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerStarted","Data":"58d40b63b82510a9f4dfceb01f6970ea4da259cda708a9384f6ccc4092a0dd64"}
Jul 16 22:46:44.891063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.891006    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" podStartSLOduration=1.890987466 podStartE2EDuration="1.890987466s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:44.874034654 +0000 UTC m=+3592.203474555" watchObservedRunningTime="2026-07-16 22:46:44.890987466 +0000 UTC m=+3592.220427370"
Jul 16 22:46:44.988026 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:44.988004    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:45.111487 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.111417    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca\") pod \"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9\" (UID: \"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9\") "
Jul 16 22:46:45.111755 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.111734    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" (UID: "b76bd3b9-1c84-4834-9bc7-0cd27800dfc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:45.212980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.212949    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:45.865452 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.865424    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"
Jul 16 22:46:45.865987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.865461    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r" event={"ID":"b76bd3b9-1c84-4834-9bc7-0cd27800dfc9","Type":"ContainerDied","Data":"cd97cfe4be6334427083a2f9d1be4183340628dd531e4cffcbf63593336d196b"}
Jul 16 22:46:45.865987 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.865521    2581 scope.go:117] "RemoveContainer" containerID="7223a270c3e57ec13fa37c4d5dfb836592ad6a73382f93e430971b3a4548237f"
Jul 16 22:46:45.867280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.867204    2581 generic.go:356] "Generic (PLEG): container finished" podID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerID="f24c83cb7f9f1f139743779700e5140a5971d212e24d05b557bccdf7ac3988a1" exitCode=6
Jul 16 22:46:45.867418 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.867289    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerDied","Data":"f24c83cb7f9f1f139743779700e5140a5971d212e24d05b557bccdf7ac3988a1"}
Jul 16 22:46:45.867513 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.867490    2581 scope.go:117] "RemoveContainer" containerID="f24c83cb7f9f1f139743779700e5140a5971d212e24d05b557bccdf7ac3988a1"
Jul 16 22:46:45.881433 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.881407    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:45.886646 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:45.886625    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-tcg5r"]
Jul 16 22:46:46.154534 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.154462    2581 scope.go:117] "RemoveContainer" containerID="76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4"
Jul 16 22:46:46.873596 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.873566    2581 generic.go:356] "Generic (PLEG): container finished" podID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerID="67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc" exitCode=6
Jul 16 22:46:46.874013 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.873680    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerDied","Data":"67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc"}
Jul 16 22:46:46.874013 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.873741    2581 scope.go:117] "RemoveContainer" containerID="f24c83cb7f9f1f139743779700e5140a5971d212e24d05b557bccdf7ac3988a1"
Jul 16 22:46:46.874013 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.873819    2581 scope.go:117] "RemoveContainer" containerID="67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc"
Jul 16 22:46:46.874172 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:46.874025    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-cbc4w_odh-ai-gateway-infra(541ae4f9-7440-4d96-ba32-1c70bd270702)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702"
Jul 16 22:46:46.875646 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.875627    2581 generic.go:356] "Generic (PLEG): container finished" podID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerID="637fca02a0f9f3e6acce9101273f32ddbb5fe52104af7ae979eaab9fe674afed" exitCode=6
Jul 16 22:46:46.875756 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.875695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" event={"ID":"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8","Type":"ContainerDied","Data":"637fca02a0f9f3e6acce9101273f32ddbb5fe52104af7ae979eaab9fe674afed"}
Jul 16 22:46:46.875963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.875948    2581 scope.go:117] "RemoveContainer" containerID="637fca02a0f9f3e6acce9101273f32ddbb5fe52104af7ae979eaab9fe674afed"
Jul 16 22:46:46.876151 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:46.876131    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw_odh-ai-gateway-infra(d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8"
Jul 16 22:46:46.884427 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:46.884411    2581 scope.go:117] "RemoveContainer" containerID="76287828b15bf2048dd841adcff1e54d18a38c5dea3e320fc9fc727bebe75ed4"
Jul 16 22:46:47.160014 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:47.159979    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" path="/var/lib/kubelet/pods/b76bd3b9-1c84-4834-9bc7-0cd27800dfc9/volumes"
Jul 16 22:46:47.883630 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:47.883603    2581 scope.go:117] "RemoveContainer" containerID="67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc"
Jul 16 22:46:47.884067 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:47.883781    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-cbc4w_odh-ai-gateway-infra(541ae4f9-7440-4d96-ba32-1c70bd270702)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702"
Jul 16 22:46:47.952435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:47.952406    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:48.093106 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.093080    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:48.237910 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.237831    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca\") pod \"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8\" (UID: \"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8\") "
Jul 16 22:46:48.238197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.238175    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" (UID: "d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:48.339285 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.339255    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:48.892097 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.892067    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"
Jul 16 22:46:48.892563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.892110    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw" event={"ID":"d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8","Type":"ContainerDied","Data":"984cf7bb8c90696a5ca5351d618e6ed6cdeff21505d4a8d56e77c81889ec5ee5"}
Jul 16 22:46:48.892563 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.892170    2581 scope.go:117] "RemoveContainer" containerID="637fca02a0f9f3e6acce9101273f32ddbb5fe52104af7ae979eaab9fe674afed"
Jul 16 22:46:48.919012 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.918984    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:48.922947 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:48.922927    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-w2npw"]
Jul 16 22:46:49.160161 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:49.160084    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" path="/var/lib/kubelet/pods/d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8/volumes"
Jul 16 22:46:52.155424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:52.155393    2581 scope.go:117] "RemoveContainer" containerID="6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b"
Jul 16 22:46:52.914624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:52.914594    2581 generic.go:356] "Generic (PLEG): container finished" podID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerID="ab418341d584d669e5a1abd0edc734027821e0b4acbfcdf4f0f85a0db324bc67" exitCode=6
Jul 16 22:46:52.914813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:52.914671    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" event={"ID":"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4","Type":"ContainerDied","Data":"ab418341d584d669e5a1abd0edc734027821e0b4acbfcdf4f0f85a0db324bc67"}
Jul 16 22:46:52.914813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:52.914743    2581 scope.go:117] "RemoveContainer" containerID="6f77bb5429bfe8973d56cfdbcbc10d2ed027b7df0c39533ebbc536c1ac43887b"
Jul 16 22:46:52.915035 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:52.915021    2581 scope.go:117] "RemoveContainer" containerID="ab418341d584d669e5a1abd0edc734027821e0b4acbfcdf4f0f85a0db324bc67"
Jul 16 22:46:52.915273 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:46:52.915251    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf_odh-ai-gateway-infra(e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4"
Jul 16 22:46:53.941004 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:53.940969    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:54.065862 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.065828    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:54.190727 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.190695    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca\") pod \"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4\" (UID: \"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4\") "
Jul 16 22:46:54.191044 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.191024    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" (UID: "e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:54.291303 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.291275    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:46:54.564124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.564100    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:46:54.595689 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.595667    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-79rh6_7befa2ae-6b26-46a9-a769-de9e01d7c4b3/ovn-acl-logging/0.log"
Jul 16 22:46:54.932645 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.932617    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"
Jul 16 22:46:54.932784 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.932655    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf" event={"ID":"e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4","Type":"ContainerDied","Data":"f02273e9014294458defb9366c487cb5c255247f034226e9bdcc31590e0d03d3"}
Jul 16 22:46:54.932784 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.932708    2581 scope.go:117] "RemoveContainer" containerID="ab418341d584d669e5a1abd0edc734027821e0b4acbfcdf4f0f85a0db324bc67"
Jul 16 22:46:54.953749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.953722    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:54.962090 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:54.962066    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-7ktxf"]
Jul 16 22:46:55.159916 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:55.159884    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" path="/var/lib/kubelet/pods/e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4/volumes"
Jul 16 22:46:59.412071 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:46:59.412032    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:47:00.154547 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:00.154518    2581 scope.go:117] "RemoveContainer" containerID="67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc"
Jul 16 22:47:00.962473 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:00.962441    2581 generic.go:356] "Generic (PLEG): container finished" podID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerID="a0720188558154d13c1a9a26978afa698f4d9648961c6838611d1bd97d2cb890" exitCode=6
Jul 16 22:47:00.962806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:00.962520    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerDied","Data":"a0720188558154d13c1a9a26978afa698f4d9648961c6838611d1bd97d2cb890"}
Jul 16 22:47:00.962806 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:00.962579    2581 scope.go:117] "RemoveContainer" containerID="67995c0dc2294fff9ab72876e71720a5250359cd374d0aba83e3410ec1c7b2dc"
Jul 16 22:47:00.962914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:00.962858    2581 scope.go:117] "RemoveContainer" containerID="a0720188558154d13c1a9a26978afa698f4d9648961c6838611d1bd97d2cb890"
Jul 16 22:47:00.963073 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:00.963053    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-cbc4w_odh-ai-gateway-infra(541ae4f9-7440-4d96-ba32-1c70bd270702)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702"
Jul 16 22:47:01.994005 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:01.993974    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:47:02.120380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.120355    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:47:02.260911 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.260830    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca\") pod \"541ae4f9-7440-4d96-ba32-1c70bd270702\" (UID: \"541ae4f9-7440-4d96-ba32-1c70bd270702\") "
Jul 16 22:47:02.261185 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.261165    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "541ae4f9-7440-4d96-ba32-1c70bd270702" (UID: "541ae4f9-7440-4d96-ba32-1c70bd270702"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:02.361720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.361695    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/541ae4f9-7440-4d96-ba32-1c70bd270702-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:47:02.974558 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.974526    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"
Jul 16 22:47:02.974699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.974522    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w" event={"ID":"541ae4f9-7440-4d96-ba32-1c70bd270702","Type":"ContainerDied","Data":"58d40b63b82510a9f4dfceb01f6970ea4da259cda708a9384f6ccc4092a0dd64"}
Jul 16 22:47:02.974699 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.974668    2581 scope.go:117] "RemoveContainer" containerID="a0720188558154d13c1a9a26978afa698f4d9648961c6838611d1bd97d2cb890"
Jul 16 22:47:02.998027 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:02.998004    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:47:03.004480 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:03.004457    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-cbc4w"]
Jul 16 22:47:03.160096 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:03.159994    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" path="/var/lib/kubelet/pods/541ae4f9-7440-4d96-ba32-1c70bd270702/volumes"
Jul 16 22:47:14.925842 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:14.925809    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:47:25.540167 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540134    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540454    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540464    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540475    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540481    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540488    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540492    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540498    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540503    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540510    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540515    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540526    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540532    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540538    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540542    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540553    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540557    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540563    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540570    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540615    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540624    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540630    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540636    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540641    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540646    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540651    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540657    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.540651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.540662    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="b76bd3b9-1c84-4834-9bc7-0cd27800dfc9" containerName="revoke-keys"
Jul 16 22:47:25.543778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.543759    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:25.546617 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.546597    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:47:25.546745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.546728    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:47:25.551760 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.551739    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:25.633909 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.633877    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq\" (UID: \"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:25.734776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.734745    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq\" (UID: \"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:25.735358 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.735329    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq\" (UID: \"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:25.854924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.854898    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:25.981933 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:25.981909    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:25.984811 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:47:25.984785    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8e7cacc_faeb_4f5a_af72_5fb2d30122f5.slice/crio-fc62e70d6ca523c60c80bb6ac3a8c55fcbdf0932b0735dec9c18efc85fdb19b8 WatchSource:0}: Error finding container fc62e70d6ca523c60c80bb6ac3a8c55fcbdf0932b0735dec9c18efc85fdb19b8: Status 404 returned error can't find the container with id fc62e70d6ca523c60c80bb6ac3a8c55fcbdf0932b0735dec9c18efc85fdb19b8
Jul 16 22:47:26.082745 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.082711    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" event={"ID":"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5","Type":"ContainerStarted","Data":"fc62e70d6ca523c60c80bb6ac3a8c55fcbdf0932b0735dec9c18efc85fdb19b8"}
Jul 16 22:47:26.333134 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333104    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:26.333455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333443    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:26.333504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333456    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:26.333504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333469    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:26.333504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333476    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:26.333504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333488    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:26.333504 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333493    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:26.333662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333541    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="e77b07c4-fba4-4ad2-ab8a-a3a86adefcb4" containerName="revoke-keys"
Jul 16 22:47:26.333662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333549    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4b1a41f-1f2d-4e3d-ada1-74ca9419b2e8" containerName="revoke-keys"
Jul 16 22:47:26.333662 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.333556    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="541ae4f9-7440-4d96-ba32-1c70bd270702" containerName="revoke-keys"
Jul 16 22:47:26.336421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.336402    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:26.345053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.345028    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:26.440750 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.440678    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7\" (UID: \"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:26.542118 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.542091    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7\" (UID: \"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:26.542804 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.542783    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7\" (UID: \"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:26.647915 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.647891    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:26.782210 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:26.782180    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:26.784101 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:47:26.784074    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa13fd4d_e4d2_4245_a8c0_a1bfeb192ef9.slice/crio-c960c50636795a565f3a21f2a8c5cd78b25b7bfaf0052527a4c33a9fd4ef3359 WatchSource:0}: Error finding container c960c50636795a565f3a21f2a8c5cd78b25b7bfaf0052527a4c33a9fd4ef3359: Status 404 returned error can't find the container with id c960c50636795a565f3a21f2a8c5cd78b25b7bfaf0052527a4c33a9fd4ef3359
Jul 16 22:47:27.094057 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.094019    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerID="d68a49e7467fa5d7f3d192636ffb21e6fef03872e1a02ec5f2c8092a311a1615" exitCode=6
Jul 16 22:47:27.094240 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.094108    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" event={"ID":"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5","Type":"ContainerDied","Data":"d68a49e7467fa5d7f3d192636ffb21e6fef03872e1a02ec5f2c8092a311a1615"}
Jul 16 22:47:27.094318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.094263    2581 scope.go:117] "RemoveContainer" containerID="d68a49e7467fa5d7f3d192636ffb21e6fef03872e1a02ec5f2c8092a311a1615"
Jul 16 22:47:27.095762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.095740    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerStarted","Data":"d692c885817619148065f8af2a2c6f44d9b561546ef92082a847d9c6ceb72d33"}
Jul 16 22:47:27.095861 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.095783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerStarted","Data":"c960c50636795a565f3a21f2a8c5cd78b25b7bfaf0052527a4c33a9fd4ef3359"}
Jul 16 22:47:27.126958 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.126917    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" podStartSLOduration=1.126904218 podStartE2EDuration="1.126904218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:47:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:47:27.125011933 +0000 UTC m=+3634.454451834" watchObservedRunningTime="2026-07-16 22:47:27.126904218 +0000 UTC m=+3634.456344153"
Jul 16 22:47:27.983005 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.982971    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:27.986278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.986259    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:27.997627 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:27.997528    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:28.054882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.054854    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l\" (UID: \"3f16c0d2-ee90-40bd-88fa-4a6da968969f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:28.102876 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.102842    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerID="34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03" exitCode=6
Jul 16 22:47:28.103021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.102932    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" event={"ID":"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5","Type":"ContainerDied","Data":"34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03"}
Jul 16 22:47:28.103021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.102987    2581 scope.go:117] "RemoveContainer" containerID="d68a49e7467fa5d7f3d192636ffb21e6fef03872e1a02ec5f2c8092a311a1615"
Jul 16 22:47:28.103150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.103131    2581 scope.go:117] "RemoveContainer" containerID="34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03"
Jul 16 22:47:28.103409 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:28.103382    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq_odh-ai-gateway-infra(f8e7cacc-faeb-4f5a-af72-5fb2d30122f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5"
Jul 16 22:47:28.104527 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.104504    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerID="d692c885817619148065f8af2a2c6f44d9b561546ef92082a847d9c6ceb72d33" exitCode=6
Jul 16 22:47:28.104600 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.104575    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerDied","Data":"d692c885817619148065f8af2a2c6f44d9b561546ef92082a847d9c6ceb72d33"}
Jul 16 22:47:28.104686 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.104673    2581 scope.go:117] "RemoveContainer" containerID="d692c885817619148065f8af2a2c6f44d9b561546ef92082a847d9c6ceb72d33"
Jul 16 22:47:28.156084 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.156057    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l\" (UID: \"3f16c0d2-ee90-40bd-88fa-4a6da968969f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:28.156656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.156637    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l\" (UID: \"3f16c0d2-ee90-40bd-88fa-4a6da968969f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:28.296410 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.296347    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:28.429000 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.428967    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:28.431808 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:47:28.431779    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f16c0d2_ee90_40bd_88fa_4a6da968969f.slice/crio-99986d3df2e03ba893b92b4f3eac473145c7cec9b3290ac6c1a0ee1f45d9c64d WatchSource:0}: Error finding container 99986d3df2e03ba893b92b4f3eac473145c7cec9b3290ac6c1a0ee1f45d9c64d: Status 404 returned error can't find the container with id 99986d3df2e03ba893b92b4f3eac473145c7cec9b3290ac6c1a0ee1f45d9c64d
Jul 16 22:47:28.906786 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:28.906747    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:47:29.112404 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.112373    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerID="7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea" exitCode=6
Jul 16 22:47:29.112850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.112457    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerDied","Data":"7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea"}
Jul 16 22:47:29.112850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.112515    2581 scope.go:117] "RemoveContainer" containerID="d692c885817619148065f8af2a2c6f44d9b561546ef92082a847d9c6ceb72d33"
Jul 16 22:47:29.112850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.112633    2581 scope.go:117] "RemoveContainer" containerID="7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea"
Jul 16 22:47:29.113016 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:29.112852    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7_odh-ai-gateway-infra(fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9"
Jul 16 22:47:29.114374 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.114352    2581 scope.go:117] "RemoveContainer" containerID="34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03"
Jul 16 22:47:29.114551 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:29.114532    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq_odh-ai-gateway-infra(f8e7cacc-faeb-4f5a-af72-5fb2d30122f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5"
Jul 16 22:47:29.115565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.115546    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerStarted","Data":"eaa7fd91c0ab3eb71baa3f89f1c9f5697db974529c714d37d07a6fdb1dcc3222"}
Jul 16 22:47:29.115667 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.115586    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerStarted","Data":"99986d3df2e03ba893b92b4f3eac473145c7cec9b3290ac6c1a0ee1f45d9c64d"}
Jul 16 22:47:29.166562 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:29.166515    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" podStartSLOduration=2.16649922 podStartE2EDuration="2.16649922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:47:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:47:29.164314512 +0000 UTC m=+3636.493754416" watchObservedRunningTime="2026-07-16 22:47:29.16649922 +0000 UTC m=+3636.495939121"
Jul 16 22:47:30.122502 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:30.122473    2581 scope.go:117] "RemoveContainer" containerID="7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea"
Jul 16 22:47:30.122976 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:30.122730    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7_odh-ai-gateway-infra(fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9"
Jul 16 22:47:30.129717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:30.129685    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerID="eaa7fd91c0ab3eb71baa3f89f1c9f5697db974529c714d37d07a6fdb1dcc3222" exitCode=6
Jul 16 22:47:30.129888 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:30.129850    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerDied","Data":"eaa7fd91c0ab3eb71baa3f89f1c9f5697db974529c714d37d07a6fdb1dcc3222"}
Jul 16 22:47:30.130109 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:30.130093    2581 scope.go:117] "RemoveContainer" containerID="eaa7fd91c0ab3eb71baa3f89f1c9f5697db974529c714d37d07a6fdb1dcc3222"
Jul 16 22:47:31.137060 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:31.136975    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerID="103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8" exitCode=6
Jul 16 22:47:31.137510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:31.137063    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerDied","Data":"103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8"}
Jul 16 22:47:31.137510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:31.137127    2581 scope.go:117] "RemoveContainer" containerID="eaa7fd91c0ab3eb71baa3f89f1c9f5697db974529c714d37d07a6fdb1dcc3222"
Jul 16 22:47:31.137510 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:31.137317    2581 scope.go:117] "RemoveContainer" containerID="103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8"
Jul 16 22:47:31.137638 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:31.137544    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l_odh-ai-gateway-infra(3f16c0d2-ee90-40bd-88fa-4a6da968969f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f"
Jul 16 22:47:32.143605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:32.143573    2581 scope.go:117] "RemoveContainer" containerID="103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8"
Jul 16 22:47:32.144094 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:32.143726    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l_odh-ai-gateway-infra(3f16c0d2-ee90-40bd-88fa-4a6da968969f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f"
Jul 16 22:47:33.931913 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:33.931882    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:33.936594 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:33.936575    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:33.942420 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:33.942393    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:34.007551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:34.007520    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-q98nt\" (UID: \"eba65322-5fe5-4949-8492-5822de33740e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:34.108278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:34.108243    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-q98nt\" (UID: \"eba65322-5fe5-4949-8492-5822de33740e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:34.108812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:34.108792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-q98nt\" (UID: \"eba65322-5fe5-4949-8492-5822de33740e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:34.248055 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:34.247980    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:34.375224 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:34.375196    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:34.377280 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:47:34.377254    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeba65322_5fe5_4949_8492_5822de33740e.slice/crio-c21a7ebab6be279d193c67c0ad17347c9cd2dc1d6213f66248bf4c0b7de479fb WatchSource:0}: Error finding container c21a7ebab6be279d193c67c0ad17347c9cd2dc1d6213f66248bf4c0b7de479fb: Status 404 returned error can't find the container with id c21a7ebab6be279d193c67c0ad17347c9cd2dc1d6213f66248bf4c0b7de479fb
Jul 16 22:47:35.160924 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:35.160894    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba65322-5fe5-4949-8492-5822de33740e" containerID="bcffc10ea8e14a273d7b77bab46bbaa87b84a7df0123424cf630e7811b932754" exitCode=6
Jul 16 22:47:35.161339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:35.160969    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" event={"ID":"eba65322-5fe5-4949-8492-5822de33740e","Type":"ContainerDied","Data":"bcffc10ea8e14a273d7b77bab46bbaa87b84a7df0123424cf630e7811b932754"}
Jul 16 22:47:35.161339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:35.161013    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" event={"ID":"eba65322-5fe5-4949-8492-5822de33740e","Type":"ContainerStarted","Data":"c21a7ebab6be279d193c67c0ad17347c9cd2dc1d6213f66248bf4c0b7de479fb"}
Jul 16 22:47:35.161339 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:35.161090    2581 scope.go:117] "RemoveContainer" containerID="bcffc10ea8e14a273d7b77bab46bbaa87b84a7df0123424cf630e7811b932754"
Jul 16 22:47:36.168124 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:36.168076    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba65322-5fe5-4949-8492-5822de33740e" containerID="6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6" exitCode=6
Jul 16 22:47:36.168565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:36.168154    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" event={"ID":"eba65322-5fe5-4949-8492-5822de33740e","Type":"ContainerDied","Data":"6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6"}
Jul 16 22:47:36.168565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:36.168249    2581 scope.go:117] "RemoveContainer" containerID="bcffc10ea8e14a273d7b77bab46bbaa87b84a7df0123424cf630e7811b932754"
Jul 16 22:47:36.168565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:36.168446    2581 scope.go:117] "RemoveContainer" containerID="6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6"
Jul 16 22:47:36.168700 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:36.168629    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-q98nt_odh-ai-gateway-infra(eba65322-5fe5-4949-8492-5822de33740e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" podUID="eba65322-5fe5-4949-8492-5822de33740e"
Jul 16 22:47:37.174191 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:37.174158    2581 scope.go:117] "RemoveContainer" containerID="6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6"
Jul 16 22:47:37.174686 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:37.174353    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-q98nt_odh-ai-gateway-infra(eba65322-5fe5-4949-8492-5822de33740e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" podUID="eba65322-5fe5-4949-8492-5822de33740e"
Jul 16 22:47:43.156914 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:43.156884    2581 scope.go:117] "RemoveContainer" containerID="34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03"
Jul 16 22:47:44.154882 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.154854    2581 scope.go:117] "RemoveContainer" containerID="103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8"
Jul 16 22:47:44.155048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.154929    2581 scope.go:117] "RemoveContainer" containerID="7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea"
Jul 16 22:47:44.209278 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.209252    2581 generic.go:356] "Generic (PLEG): container finished" podID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerID="6abdfbd4a31bdfc7731148f3bcd8b86fb4bd0feaacbf93e3f9c3d21ecbcaa594" exitCode=6
Jul 16 22:47:44.209628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.209331    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" event={"ID":"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5","Type":"ContainerDied","Data":"6abdfbd4a31bdfc7731148f3bcd8b86fb4bd0feaacbf93e3f9c3d21ecbcaa594"}
Jul 16 22:47:44.209628 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.209407    2581 scope.go:117] "RemoveContainer" containerID="34f121fcd229a6689ae3962766dfd63d4020fa9d405334f2d7f8c347b85d9d03"
Jul 16 22:47:44.209736 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:44.209693    2581 scope.go:117] "RemoveContainer" containerID="6abdfbd4a31bdfc7731148f3bcd8b86fb4bd0feaacbf93e3f9c3d21ecbcaa594"
Jul 16 22:47:44.209888 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:44.209867    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq_odh-ai-gateway-infra(f8e7cacc-faeb-4f5a-af72-5fb2d30122f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5"
Jul 16 22:47:45.217865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.217773    2581 generic.go:356] "Generic (PLEG): container finished" podID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerID="41521ddc4e1a76faba0f162f58ca57398ffa810b656703bf7815eec1f1da5591" exitCode=6
Jul 16 22:47:45.217865 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.217847    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerDied","Data":"41521ddc4e1a76faba0f162f58ca57398ffa810b656703bf7815eec1f1da5591"}
Jul 16 22:47:45.218385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.217907    2581 scope.go:117] "RemoveContainer" containerID="103bce682db8c7de8b32355cfff0017ba5c3d3456d9c89f455a1211283c6f0a8"
Jul 16 22:47:45.218385 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.218218    2581 scope.go:117] "RemoveContainer" containerID="41521ddc4e1a76faba0f162f58ca57398ffa810b656703bf7815eec1f1da5591"
Jul 16 22:47:45.218495 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:45.218472    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l_odh-ai-gateway-infra(3f16c0d2-ee90-40bd-88fa-4a6da968969f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f"
Jul 16 22:47:45.219767 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.219743    2581 generic.go:356] "Generic (PLEG): container finished" podID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerID="4a083b203356f2f1027b51d8e3883594e5dd4c2536bbf4613e6a11612fc51bbd" exitCode=6
Jul 16 22:47:45.219886 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.219815    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerDied","Data":"4a083b203356f2f1027b51d8e3883594e5dd4c2536bbf4613e6a11612fc51bbd"}
Jul 16 22:47:45.220083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.220068    2581 scope.go:117] "RemoveContainer" containerID="4a083b203356f2f1027b51d8e3883594e5dd4c2536bbf4613e6a11612fc51bbd"
Jul 16 22:47:45.220280 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:45.220259    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7_odh-ai-gateway-infra(fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9"
Jul 16 22:47:45.229211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.229194    2581 scope.go:117] "RemoveContainer" containerID="7ab0f3eb5df0cca1b81ec0d2adacaa359900ece2a7f576534fab671df290e2ea"
Jul 16 22:47:45.249194 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.249163    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:45.388094 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.388065    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:45.512028 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.511947    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca\") pod \"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5\" (UID: \"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5\") "
Jul 16 22:47:45.512309 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.512281    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" (UID: "f8e7cacc-faeb-4f5a-af72-5fb2d30122f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:45.612516 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:45.612475    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:47:46.225388 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.225358    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"
Jul 16 22:47:46.225388 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.225378    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq" event={"ID":"f8e7cacc-faeb-4f5a-af72-5fb2d30122f5","Type":"ContainerDied","Data":"fc62e70d6ca523c60c80bb6ac3a8c55fcbdf0932b0735dec9c18efc85fdb19b8"}
Jul 16 22:47:46.225896 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.225444    2581 scope.go:117] "RemoveContainer" containerID="6abdfbd4a31bdfc7731148f3bcd8b86fb4bd0feaacbf93e3f9c3d21ecbcaa594"
Jul 16 22:47:46.247370 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.247333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:46.254576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.254556    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:46.259434 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.259413    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-ds2wq"]
Jul 16 22:47:46.287812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.287778    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:46.409280 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.409255    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:46.440546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.440516    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:46.519816 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.519726    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca\") pod \"3f16c0d2-ee90-40bd-88fa-4a6da968969f\" (UID: \"3f16c0d2-ee90-40bd-88fa-4a6da968969f\") "
Jul 16 22:47:46.519816 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.519801    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca\") pod \"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9\" (UID: \"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9\") "
Jul 16 22:47:46.520119 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.520095    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" (UID: "fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:46.520184 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.520125    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3f16c0d2-ee90-40bd-88fa-4a6da968969f" (UID: "3f16c0d2-ee90-40bd-88fa-4a6da968969f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:46.621089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.621049    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:47:46.621089 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:46.621083    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f16c0d2-ee90-40bd-88fa-4a6da968969f-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:47:47.160921 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.160886    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" path="/var/lib/kubelet/pods/f8e7cacc-faeb-4f5a-af72-5fb2d30122f5/volumes"
Jul 16 22:47:47.235803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.235771    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"
Jul 16 22:47:47.235803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.235783    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l" event={"ID":"3f16c0d2-ee90-40bd-88fa-4a6da968969f","Type":"ContainerDied","Data":"99986d3df2e03ba893b92b4f3eac473145c7cec9b3290ac6c1a0ee1f45d9c64d"}
Jul 16 22:47:47.236353 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.235850    2581 scope.go:117] "RemoveContainer" containerID="41521ddc4e1a76faba0f162f58ca57398ffa810b656703bf7815eec1f1da5591"
Jul 16 22:47:47.237275 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.237249    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7" event={"ID":"fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9","Type":"ContainerDied","Data":"c960c50636795a565f3a21f2a8c5cd78b25b7bfaf0052527a4c33a9fd4ef3359"}
Jul 16 22:47:47.237275 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.237266    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"
Jul 16 22:47:47.244354 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.244330    2581 scope.go:117] "RemoveContainer" containerID="4a083b203356f2f1027b51d8e3883594e5dd4c2536bbf4613e6a11612fc51bbd"
Jul 16 22:47:47.258613 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.258586    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:47.264241 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.264204    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-lhc6l"]
Jul 16 22:47:47.281598 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.281554    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:47.283570 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:47.283546    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-4d06d9c2-zs6t7"]
Jul 16 22:47:49.159975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:49.159942    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" path="/var/lib/kubelet/pods/3f16c0d2-ee90-40bd-88fa-4a6da968969f/volumes"
Jul 16 22:47:49.160356 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:49.160290    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" path="/var/lib/kubelet/pods/fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9/volumes"
Jul 16 22:47:52.155306 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:52.155274    2581 scope.go:117] "RemoveContainer" containerID="6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6"
Jul 16 22:47:53.268470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.268434    2581 generic.go:356] "Generic (PLEG): container finished" podID="eba65322-5fe5-4949-8492-5822de33740e" containerID="0dbf3bc3f17574a2e3f2666c4138eca29612887638710030e65e53d87a12554b" exitCode=6
Jul 16 22:47:53.268881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.268504    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" event={"ID":"eba65322-5fe5-4949-8492-5822de33740e","Type":"ContainerDied","Data":"0dbf3bc3f17574a2e3f2666c4138eca29612887638710030e65e53d87a12554b"}
Jul 16 22:47:53.268881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.268569    2581 scope.go:117] "RemoveContainer" containerID="6f06291188535491ddbf19f7fc18671c8cb3d2285c49f1eb54ca063dc5b6d7f6"
Jul 16 22:47:53.268881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.268866    2581 scope.go:117] "RemoveContainer" containerID="0dbf3bc3f17574a2e3f2666c4138eca29612887638710030e65e53d87a12554b"
Jul 16 22:47:53.269143 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:47:53.269111    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-q98nt_odh-ai-gateway-infra(eba65322-5fe5-4949-8492-5822de33740e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" podUID="eba65322-5fe5-4949-8492-5822de33740e"
Jul 16 22:47:53.828729 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.828699    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"]
Jul 16 22:47:53.833439 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:53.833411    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-fbed9b-29737365-9nvg7"]
Jul 16 22:47:54.298940 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:54.298894    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:54.431807 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:54.431786    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:54.584387 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:54.584310    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca\") pod \"eba65322-5fe5-4949-8492-5822de33740e\" (UID: \"eba65322-5fe5-4949-8492-5822de33740e\") "
Jul 16 22:47:54.584687 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:54.584667    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "eba65322-5fe5-4949-8492-5822de33740e" (UID: "eba65322-5fe5-4949-8492-5822de33740e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:47:54.685372 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:54.685344    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/eba65322-5fe5-4949-8492-5822de33740e-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:47:55.160884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.160851    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95a134b0-ecab-47be-8243-1cbe1947933f" path="/var/lib/kubelet/pods/95a134b0-ecab-47be-8243-1cbe1947933f/volumes"
Jul 16 22:47:55.280854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.280826    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"
Jul 16 22:47:55.280854 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.280838    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt" event={"ID":"eba65322-5fe5-4949-8492-5822de33740e","Type":"ContainerDied","Data":"c21a7ebab6be279d193c67c0ad17347c9cd2dc1d6213f66248bf4c0b7de479fb"}
Jul 16 22:47:55.281078 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.280891    2581 scope.go:117] "RemoveContainer" containerID="0dbf3bc3f17574a2e3f2666c4138eca29612887638710030e65e53d87a12554b"
Jul 16 22:47:55.302740 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.302717    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:55.306834 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:55.306814    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-q98nt"]
Jul 16 22:47:57.160769 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:57.160736    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eba65322-5fe5-4949-8492-5822de33740e" path="/var/lib/kubelet/pods/eba65322-5fe5-4949-8492-5822de33740e/volumes"
Jul 16 22:47:57.305368 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:47:57.305336    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:48:09.130980 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:09.130942    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:48:09.131460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:09.131241    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy" containerID="cri-o://05496b8116e703d11d34a0ab39b433e10c01ae88e23f5dbdb29a01aed78b88b5" gracePeriod=30
Jul 16 22:48:14.369391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.369350    2581 generic.go:356] "Generic (PLEG): container finished" podID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerID="05496b8116e703d11d34a0ab39b433e10c01ae88e23f5dbdb29a01aed78b88b5" exitCode=0
Jul 16 22:48:14.369812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.369387    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerDied","Data":"05496b8116e703d11d34a0ab39b433e10c01ae88e23f5dbdb29a01aed78b88b5"}
Jul 16 22:48:14.369812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.369466    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj" event={"ID":"781cea4b-97d1-4af5-9ee2-6b8e41596578","Type":"ContainerDied","Data":"d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b"}
Jul 16 22:48:14.369812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.369493    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1444b18483f2b2bcd1f5239a794736a32f13dbdbf83953ad59c5c093fb4df9b"
Jul 16 22:48:14.369812 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.369515    2581 scope.go:117] "RemoveContainer" containerID="eebe4d24abc66f1bb57d1af607dd5c9c5fc2ca5c471f68418fb1d8b795ceb095"
Jul 16 22:48:14.386757 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.386734    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:48:14.455634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455560    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455599    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455634 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455619    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455646    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455772    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455825    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.455898 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455860    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.456137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455912    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.456137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455919    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:14.456137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.455954    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs\") pod \"781cea4b-97d1-4af5-9ee2-6b8e41596578\" (UID: \"781cea4b-97d1-4af5-9ee2-6b8e41596578\") "
Jul 16 22:48:14.456137 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456009    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:14.456546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456126    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:14.456546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456222    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.456546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456257    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:14.456546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456266    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/781cea4b-97d1-4af5-9ee2-6b8e41596578-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.456546 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456280    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.457003 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.456973    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data" (OuterVolumeSpecName: "istio-data") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:14.457970 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.457942    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token" (OuterVolumeSpecName: "istio-token") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:14.458063 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.457988    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:14.458102 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.458077    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:48:14.458590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.458567    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp" (OuterVolumeSpecName: "kube-api-access-wvlzp") pod "781cea4b-97d1-4af5-9ee2-6b8e41596578" (UID: "781cea4b-97d1-4af5-9ee2-6b8e41596578"). InnerVolumeSpecName "kube-api-access-wvlzp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:14.557379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557355    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.557379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557374    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.557379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557383    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.557551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557391    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/781cea4b-97d1-4af5-9ee2-6b8e41596578-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.557551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557400    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:14.557551 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:14.557408    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvlzp\" (UniqueName: \"kubernetes.io/projected/781cea4b-97d1-4af5-9ee2-6b8e41596578-kube-api-access-wvlzp\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:15.374985 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:15.374954    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"
Jul 16 22:48:15.392189 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:15.392158    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:48:15.396526 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:15.396496    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-fbed9b-openshift-default-8646846bc5-px8hj"]
Jul 16 22:48:17.159781 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:17.159745    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" path="/var/lib/kubelet/pods/781cea4b-97d1-4af5-9ee2-6b8e41596578/volumes"
Jul 16 22:48:18.069859 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.069823    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:18.070170 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070157    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070172    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070189    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070198    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070205    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070211    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.070218 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070218    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070223    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070245    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070253    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070266    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070274    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070284    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070288    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070296    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070301    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070306    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070310    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070315    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070321    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070333    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070338    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070389    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070396    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070401    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070407    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070412    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070417    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:18.070415 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070422    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.071040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070427    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:18.071040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070436    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:18.071040 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.070446    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="781cea4b-97d1-4af5-9ee2-6b8e41596578" containerName="istio-proxy"
Jul 16 22:48:18.074673 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.074657    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:18.079424 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.079403    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:48:18.082492 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.082475    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:48:18.087150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.087133    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf\" (UID: \"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:18.093021 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.093000    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:18.187937 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.187906    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf\" (UID: \"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:18.188531 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.188511    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf\" (UID: \"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:18.383932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.383903    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:18.510351 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:18.510328    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:18.512154 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:48:18.512126    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cf5b5c1_d490_438f_a51b_9c3f762e5fc3.slice/crio-9a7a4f4482698085929bd7a60ace0b7814351bb7e81e07c906be1b0c33a0255b WatchSource:0}: Error finding container 9a7a4f4482698085929bd7a60ace0b7814351bb7e81e07c906be1b0c33a0255b: Status 404 returned error can't find the container with id 9a7a4f4482698085929bd7a60ace0b7814351bb7e81e07c906be1b0c33a0255b
Jul 16 22:48:19.395312 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:19.395273    2581 generic.go:356] "Generic (PLEG): container finished" podID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerID="ce66bd666006500e2d25a3a83f44e739d26aa535b82d1a3f2bb4353325d9bf7f" exitCode=6
Jul 16 22:48:19.395668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:19.395362    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerDied","Data":"ce66bd666006500e2d25a3a83f44e739d26aa535b82d1a3f2bb4353325d9bf7f"}
Jul 16 22:48:19.395668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:19.395416    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerStarted","Data":"9a7a4f4482698085929bd7a60ace0b7814351bb7e81e07c906be1b0c33a0255b"}
Jul 16 22:48:19.395668 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:19.395512    2581 scope.go:117] "RemoveContainer" containerID="ce66bd666006500e2d25a3a83f44e739d26aa535b82d1a3f2bb4353325d9bf7f"
Jul 16 22:48:20.082101 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.082065    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"]
Jul 16 22:48:20.087514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.087488    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-106c85-29737365-2c9nd"]
Jul 16 22:48:20.402650 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.402617    2581 generic.go:356] "Generic (PLEG): container finished" podID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerID="5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324" exitCode=6
Jul 16 22:48:20.403073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.402704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerDied","Data":"5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324"}
Jul 16 22:48:20.403073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.402764    2581 scope.go:117] "RemoveContainer" containerID="ce66bd666006500e2d25a3a83f44e739d26aa535b82d1a3f2bb4353325d9bf7f"
Jul 16 22:48:20.403073 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:20.402938    2581 scope.go:117] "RemoveContainer" containerID="5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324"
Jul 16 22:48:20.403206 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:20.403139    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf_odh-ai-gateway-infra(0cf5b5c1-d490-438f-a51b-9c3f762e5fc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3"
Jul 16 22:48:21.164411 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:21.164369    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="317006b8-7a96-4592-b2a8-48fba5c7d1d0" path="/var/lib/kubelet/pods/317006b8-7a96-4592-b2a8-48fba5c7d1d0/volumes"
Jul 16 22:48:21.409488 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:21.409458    2581 scope.go:117] "RemoveContainer" containerID="5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324"
Jul 16 22:48:21.409909 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:21.409645    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf_odh-ai-gateway-infra(0cf5b5c1-d490-438f-a51b-9c3f762e5fc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3"
Jul 16 22:48:21.818129 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:21.818094    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-87hdc"]
Jul 16 22:48:25.593807 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.593771    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:25.594153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594119    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:25.594153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594132    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:25.594153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594140    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:25.594153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594145    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:25.594153 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594153    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:25.594362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594158    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:25.594362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594221    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="f8e7cacc-faeb-4f5a-af72-5fb2d30122f5" containerName="revoke-keys"
Jul 16 22:48:25.594362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594242    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa13fd4d-e4d2-4245-a8c0-a1bfeb192ef9" containerName="revoke-keys"
Jul 16 22:48:25.594362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594247    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f16c0d2-ee90-40bd-88fa-4a6da968969f" containerName="revoke-keys"
Jul 16 22:48:25.594362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.594256    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="eba65322-5fe5-4949-8492-5822de33740e" containerName="revoke-keys"
Jul 16 22:48:25.598590 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.598573    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:25.604963 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.604933    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:25.651479 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.651450    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw\" (UID: \"da840848-57cf-47e0-9327-5df27b15e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:25.752708 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.752676    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw\" (UID: \"da840848-57cf-47e0-9327-5df27b15e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:25.753380 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.753356    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw\" (UID: \"da840848-57cf-47e0-9327-5df27b15e8ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:25.909630 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:25.909541    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:26.042555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:26.042527    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:26.044991 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:48:26.044964    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda840848_57cf_47e0_9327_5df27b15e8ff.slice/crio-fd80f7f4ade181a2806fea72dd090af47af765e354e04881071bfb64af79daeb WatchSource:0}: Error finding container fd80f7f4ade181a2806fea72dd090af47af765e354e04881071bfb64af79daeb: Status 404 returned error can't find the container with id fd80f7f4ade181a2806fea72dd090af47af765e354e04881071bfb64af79daeb
Jul 16 22:48:26.435869 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:26.435834    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerStarted","Data":"032bbf041f6cb82134779d4f2f7848a6d5bb482cf773cd4e26d97d0d2cf40e0f"}
Jul 16 22:48:26.436069 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:26.435889    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerStarted","Data":"fd80f7f4ade181a2806fea72dd090af47af765e354e04881071bfb64af79daeb"}
Jul 16 22:48:27.442435 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:27.442392    2581 generic.go:356] "Generic (PLEG): container finished" podID="da840848-57cf-47e0-9327-5df27b15e8ff" containerID="032bbf041f6cb82134779d4f2f7848a6d5bb482cf773cd4e26d97d0d2cf40e0f" exitCode=6
Jul 16 22:48:27.442905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:27.442476    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerDied","Data":"032bbf041f6cb82134779d4f2f7848a6d5bb482cf773cd4e26d97d0d2cf40e0f"}
Jul 16 22:48:27.442905 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:27.442624    2581 scope.go:117] "RemoveContainer" containerID="032bbf041f6cb82134779d4f2f7848a6d5bb482cf773cd4e26d97d0d2cf40e0f"
Jul 16 22:48:28.448476 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:28.448392    2581 generic.go:356] "Generic (PLEG): container finished" podID="da840848-57cf-47e0-9327-5df27b15e8ff" containerID="6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9" exitCode=6
Jul 16 22:48:28.448870 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:28.448472    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerDied","Data":"6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9"}
Jul 16 22:48:28.448870 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:28.448530    2581 scope.go:117] "RemoveContainer" containerID="032bbf041f6cb82134779d4f2f7848a6d5bb482cf773cd4e26d97d0d2cf40e0f"
Jul 16 22:48:28.448870 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:28.448745    2581 scope.go:117] "RemoveContainer" containerID="6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9"
Jul 16 22:48:28.448987 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:28.448956    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw_odh-ai-gateway-infra(da840848-57cf-47e0-9327-5df27b15e8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" podUID="da840848-57cf-47e0-9327-5df27b15e8ff"
Jul 16 22:48:29.455365 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:29.455332    2581 scope.go:117] "RemoveContainer" containerID="6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9"
Jul 16 22:48:29.455739 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:29.455525    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw_odh-ai-gateway-infra(da840848-57cf-47e0-9327-5df27b15e8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" podUID="da840848-57cf-47e0-9327-5df27b15e8ff"
Jul 16 22:48:33.977200 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:33.977167    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:33.980816 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:33.980794    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:33.988242 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:33.988198    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:34.132133 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.132097    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-r2hg9\" (UID: \"437ddb4b-0e45-4e57-a2f9-8357144a8d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:34.233330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.233200    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-r2hg9\" (UID: \"437ddb4b-0e45-4e57-a2f9-8357144a8d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:34.233778 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.233758    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-a6237863-r2hg9\" (UID: \"437ddb4b-0e45-4e57-a2f9-8357144a8d9d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:34.293462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.293422    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:34.427420 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.427392    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:34.429262 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:48:34.429213    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod437ddb4b_0e45_4e57_a2f9_8357144a8d9d.slice/crio-04e9b8adbd411249b5bc26f9f0997ee6066bb422d0d87ec222245158711d783b WatchSource:0}: Error finding container 04e9b8adbd411249b5bc26f9f0997ee6066bb422d0d87ec222245158711d783b: Status 404 returned error can't find the container with id 04e9b8adbd411249b5bc26f9f0997ee6066bb422d0d87ec222245158711d783b
Jul 16 22:48:34.479552 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:34.479517    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerStarted","Data":"04e9b8adbd411249b5bc26f9f0997ee6066bb422d0d87ec222245158711d783b"}
Jul 16 22:48:35.275454 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:35.275420    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:48:35.275932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:35.275669    2581 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" containerID="cri-o://47effa85f305af135d17a2db7924b40d86ec04be12fb35fd9e37005ea356c7af" gracePeriod=30
Jul 16 22:48:35.486733 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:35.486696    2581 generic.go:356] "Generic (PLEG): container finished" podID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerID="69f2ed6f9d7b7ec9f19319a92ee951cb30b3e759d63e8dd5982491a6d4524b45" exitCode=6
Jul 16 22:48:35.486899 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:35.486744    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerDied","Data":"69f2ed6f9d7b7ec9f19319a92ee951cb30b3e759d63e8dd5982491a6d4524b45"}
Jul 16 22:48:35.486967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:35.486947    2581 scope.go:117] "RemoveContainer" containerID="69f2ed6f9d7b7ec9f19319a92ee951cb30b3e759d63e8dd5982491a6d4524b45"
Jul 16 22:48:36.155247 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.155209    2581 scope.go:117] "RemoveContainer" containerID="5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324"
Jul 16 22:48:36.493728 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.493695    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerStarted","Data":"540eb7662c2b4fc8bb25201bb7be92c6dcf4c2e7cbaee6a89d628889ece5c12b"}
Jul 16 22:48:36.495514 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.495488    2581 generic.go:356] "Generic (PLEG): container finished" podID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerID="47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8" exitCode=6
Jul 16 22:48:36.495632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.495548    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerDied","Data":"47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8"}
Jul 16 22:48:36.495632 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.495612    2581 scope.go:117] "RemoveContainer" containerID="69f2ed6f9d7b7ec9f19319a92ee951cb30b3e759d63e8dd5982491a6d4524b45"
Jul 16 22:48:36.495758 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.495743    2581 scope.go:117] "RemoveContainer" containerID="47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8"
Jul 16 22:48:36.495920 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:36.495902    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-r2hg9_odh-ai-gateway-infra(437ddb4b-0e45-4e57-a2f9-8357144a8d9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d"
Jul 16 22:48:36.509639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:36.509587    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" podStartSLOduration=18.509574677 podStartE2EDuration="18.509574677s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:36.508593683 +0000 UTC m=+3703.838033591" watchObservedRunningTime="2026-07-16 22:48:36.509574677 +0000 UTC m=+3703.839014611"
Jul 16 22:48:37.501832 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.501798    2581 generic.go:356] "Generic (PLEG): container finished" podID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerID="540eb7662c2b4fc8bb25201bb7be92c6dcf4c2e7cbaee6a89d628889ece5c12b" exitCode=6
Jul 16 22:48:37.502315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.501866    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerDied","Data":"540eb7662c2b4fc8bb25201bb7be92c6dcf4c2e7cbaee6a89d628889ece5c12b"}
Jul 16 22:48:37.502315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.501931    2581 scope.go:117] "RemoveContainer" containerID="5c3c810222f8697c5a67a6883ed50304e3f664ebabddf08a30f2685435065324"
Jul 16 22:48:37.502315 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.502253    2581 scope.go:117] "RemoveContainer" containerID="540eb7662c2b4fc8bb25201bb7be92c6dcf4c2e7cbaee6a89d628889ece5c12b"
Jul 16 22:48:37.502468 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:37.502448    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf_odh-ai-gateway-infra(0cf5b5c1-d490-438f-a51b-9c3f762e5fc3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3"
Jul 16 22:48:37.503711 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.503693    2581 scope.go:117] "RemoveContainer" containerID="47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8"
Jul 16 22:48:37.503874 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:37.503855    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-r2hg9_odh-ai-gateway-infra(437ddb4b-0e45-4e57-a2f9-8357144a8d9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d"
Jul 16 22:48:37.519318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:37.519296    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:38.604813 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.604777    2581 patch_prober.go:28] interesting pod/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 22:48:38.605150 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.604839    2581 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 22:48:38.638981 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.638959    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:38.774068 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.773984    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca\") pod \"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3\" (UID: \"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3\") "
Jul 16 22:48:38.774364 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.774343    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" (UID: "0cf5b5c1-d490-438f-a51b-9c3f762e5fc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:38.875561 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:38.875530    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:39.514860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:39.514775    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf" event={"ID":"0cf5b5c1-d490-438f-a51b-9c3f762e5fc3","Type":"ContainerDied","Data":"9a7a4f4482698085929bd7a60ace0b7814351bb7e81e07c906be1b0c33a0255b"}
Jul 16 22:48:39.514860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:39.514802    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"
Jul 16 22:48:39.514860 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:39.514829    2581 scope.go:117] "RemoveContainer" containerID="540eb7662c2b4fc8bb25201bb7be92c6dcf4c2e7cbaee6a89d628889ece5c12b"
Jul 16 22:48:39.531932 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:39.531904    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:39.537311 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:39.537278    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-2nlnf"]
Jul 16 22:48:40.522379 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.522351    2581 generic.go:356] "Generic (PLEG): container finished" podID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerID="47effa85f305af135d17a2db7924b40d86ec04be12fb35fd9e37005ea356c7af" exitCode=0
Jul 16 22:48:40.522776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.522424    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerDied","Data":"47effa85f305af135d17a2db7924b40d86ec04be12fb35fd9e37005ea356c7af"}
Jul 16 22:48:40.522776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.522493    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc" event={"ID":"0c562dd1-ea43-47b8-95af-e20eb2907723","Type":"ContainerDied","Data":"edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110"}
Jul 16 22:48:40.522776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.522519    2581 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="edc8881945119988bd4cb2c6f4cf44153f9c15d23ac38b45cb3814b3f2c6e110"
Jul 16 22:48:40.522776 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.522538    2581 scope.go:117] "RemoveContainer" containerID="a803bc7e6cb9f62b581686f52664a23d14ee4a8f730dbd1d795e18f4668d3f18"
Jul 16 22:48:40.523880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.523866    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:48:40.590211 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590188    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590292    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590331    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590362 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590355    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590398    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590470 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590430    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tstqt\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590471    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590567 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590495    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.590919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590792    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:40.590919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590809    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:40.590919 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590877    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:40.591390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.590953    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert\") pod \"0c562dd1-ea43-47b8-95af-e20eb2907723\" (UID: \"0c562dd1-ea43-47b8-95af-e20eb2907723\") "
Jul 16 22:48:40.591390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.591220    2581 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.591390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.591265    2581 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-credential-socket\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.591390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.591281    2581 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-workload-certs\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.591390 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.591284    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:40.591762 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.591736    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data" (OuterVolumeSpecName: "istio-data") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:40.592678 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.592660    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:48:40.592761 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.592735    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt" (OuterVolumeSpecName: "kube-api-access-tstqt") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "kube-api-access-tstqt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:40.592959 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.592944    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token" (OuterVolumeSpecName: "istio-token") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:48:40.593025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.593013    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0c562dd1-ea43-47b8-95af-e20eb2907723" (UID: "0c562dd1-ea43-47b8-95af-e20eb2907723"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 16 22:48:40.692157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692132    2581 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-token\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.692157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692151    2581 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-envoy\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.692157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692161    2581 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tstqt\" (UniqueName: \"kubernetes.io/projected/0c562dd1-ea43-47b8-95af-e20eb2907723-kube-api-access-tstqt\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.692349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692170    2581 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-data\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.692349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692178    2581 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c562dd1-ea43-47b8-95af-e20eb2907723-istiod-ca-cert\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:40.692349 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:40.692186    2581 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c562dd1-ea43-47b8-95af-e20eb2907723-istio-podinfo\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:41.160279 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.160215    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" path="/var/lib/kubelet/pods/0cf5b5c1-d490-438f-a51b-9c3f762e5fc3/volumes"
Jul 16 22:48:41.348639 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.348612    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cgdjh_2f807aa4-2b3c-42c9-bb11-0c42de8ca13c/manager/0.log"
Jul 16 22:48:41.473193 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.473127    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z2hnh_bad5efa6-6e90-43d3-838a-d553b568c204/manager/0.log"
Jul 16 22:48:41.528903 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.528878    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"
Jul 16 22:48:41.561320 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.561295    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:48:41.567965 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.567945    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-106c85-openshift-default-765fc947dc-nwssc"]
Jul 16 22:48:41.707357 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.707326    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fgrdt_fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2/server/0.log"
Jul 16 22:48:41.950331 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.950309    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-pzzpd_a852da53-3657-47a1-b8f0-78f8dfd0985a/copy-manifests/0.log"
Jul 16 22:48:41.965880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:41.965858    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-pzzpd_a852da53-3657-47a1-b8f0-78f8dfd0985a/manager/0.log"
Jul 16 22:48:42.731143 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:42.731113    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737335-7db6w_fea86aeb-6ba0-415b-bcfa-20e0cc58c276/cleanup/0.log"
Jul 16 22:48:43.065603 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.065523    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw_da840848-57cf-47e0-9327-5df27b15e8ff/revoke-keys/1.log"
Jul 16 22:48:43.156955 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.156926    2581 scope.go:117] "RemoveContainer" containerID="6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9"
Jul 16 22:48:43.160391 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.160367    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" path="/var/lib/kubelet/pods/0c562dd1-ea43-47b8-95af-e20eb2907723/volumes"
Jul 16 22:48:43.290539 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.290511    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-24b876-5jvd7_928cf688-cc12-4150-8285-85f2ee813e41/revoke-keys/0.log"
Jul 16 22:48:43.403789 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.403766    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-a6237863-r2hg9_437ddb4b-0e45-4e57-a2f9-8357144a8d9d/revoke-keys/1.log"
Jul 16 22:48:43.510373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.510351    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-3a3e4b-vm7ks_a05fa8c5-396e-48a4-9ceb-a77a1b8a3761/revoke-keys/0.log"
Jul 16 22:48:43.540509 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.540473    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerStarted","Data":"9875deb81c1956acbe69f10697576e744248b7afe07336f8d0fa8d8a1dcb5ad9"}
Jul 16 22:48:43.559747 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.559698    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" podStartSLOduration=18.559683347 podStartE2EDuration="18.559683347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:26.471655938 +0000 UTC m=+3693.801095840" watchObservedRunningTime="2026-07-16 22:48:43.559683347 +0000 UTC m=+3710.889123248"
Jul 16 22:48:43.625749 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.625677    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-2fe960-pm7k8_9eec6cdb-e319-44db-8e5b-028294782eee/revoke-keys/0.log"
Jul 16 22:48:43.847192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.847167    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-50ab3915-2mfrh_83e480fb-3074-42ce-9d89-9dc3fdc4af92/revoke-keys/0.log"
Jul 16 22:48:43.956469 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:43.956385    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-b3fb3ce5-7z4n5_5a42b7f6-c42c-45cf-82a3-b481364fa46e/revoke-keys/0.log"
Jul 16 22:48:44.546840 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:44.546806    2581 generic.go:356] "Generic (PLEG): container finished" podID="da840848-57cf-47e0-9327-5df27b15e8ff" containerID="9875deb81c1956acbe69f10697576e744248b7afe07336f8d0fa8d8a1dcb5ad9" exitCode=6
Jul 16 22:48:44.547024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:44.546879    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerDied","Data":"9875deb81c1956acbe69f10697576e744248b7afe07336f8d0fa8d8a1dcb5ad9"}
Jul 16 22:48:44.547024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:44.546937    2581 scope.go:117] "RemoveContainer" containerID="6cdb3833f08e457812ae8c9e774d339007101ecee5744f5e4702511a37af61f9"
Jul 16 22:48:44.547299 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:44.547270    2581 scope.go:117] "RemoveContainer" containerID="9875deb81c1956acbe69f10697576e744248b7afe07336f8d0fa8d8a1dcb5ad9"
Jul 16 22:48:44.547482 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:44.547462    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw_odh-ai-gateway-infra(da840848-57cf-47e0-9327-5df27b15e8ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" podUID="da840848-57cf-47e0-9327-5df27b15e8ff"
Jul 16 22:48:44.573714 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:44.573687    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:45.685967 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:45.685943    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:45.801623 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:45.801596    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4dfh2_2cad4968-06e6-41c0-90f0-9443f79d4f44/manager/0.log"
Jul 16 22:48:45.836693 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:45.836668    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca\") pod \"da840848-57cf-47e0-9327-5df27b15e8ff\" (UID: \"da840848-57cf-47e0-9327-5df27b15e8ff\") "
Jul 16 22:48:45.837025 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:45.837006    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "da840848-57cf-47e0-9327-5df27b15e8ff" (UID: "da840848-57cf-47e0-9327-5df27b15e8ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:45.937373 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:45.937351    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da840848-57cf-47e0-9327-5df27b15e8ff-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:46.265982 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.265902    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-87hdc_c25947ed-c099-4728-b170-10980f34bf79/limitador/0.log"
Jul 16 22:48:46.383880 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.383851    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bhm6k_8a872677-006e-4af2-99cc-23f1ebaed274/manager/0.log"
Jul 16 22:48:46.558641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.558569    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"
Jul 16 22:48:46.558641 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.558597    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw" event={"ID":"da840848-57cf-47e0-9327-5df27b15e8ff","Type":"ContainerDied","Data":"fd80f7f4ade181a2806fea72dd090af47af765e354e04881071bfb64af79daeb"}
Jul 16 22:48:46.558851 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.558650    2581 scope.go:117] "RemoveContainer" containerID="9875deb81c1956acbe69f10697576e744248b7afe07336f8d0fa8d8a1dcb5ad9"
Jul 16 22:48:46.579369 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.579333    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:46.585422 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.585403    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-24cdbda3-vbgpw"]
Jul 16 22:48:46.739581 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.739555    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4m9wss_083b09d1-de49-4950-a8fa-75b4cc05a71d/istio-proxy/0.log"
Jul 16 22:48:46.867997 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:46.867970    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnq2_d287d78c-ded5-4b18-896c-c64ab048bcd3/discovery/0.log"
Jul 16 22:48:47.102936 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:47.102914    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-xhws9_30709861-ba73-4a0e-8306-5f3e1b991dee/kube-auth-proxy/0.log"
Jul 16 22:48:47.160616 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:47.160536    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" path="/var/lib/kubelet/pods/da840848-57cf-47e0-9327-5df27b15e8ff/volumes"
Jul 16 22:48:53.157715 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:53.157685    2581 scope.go:117] "RemoveContainer" containerID="47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8"
Jul 16 22:48:53.592455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:53.592423    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerStarted","Data":"d6470d35922983aa0ae42e811657a986d74d14ad880866c8ea91e2bd9c110ecd"}
Jul 16 22:48:53.610325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:53.610285    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" podStartSLOduration=20.610271843 podStartE2EDuration="20.610271843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:48:53.609061058 +0000 UTC m=+3720.938500970" watchObservedRunningTime="2026-07-16 22:48:53.610271843 +0000 UTC m=+3720.939711741"
Jul 16 22:48:53.814162 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:53.814085    2581 scope.go:117] "RemoveContainer" containerID="47effa85f305af135d17a2db7924b40d86ec04be12fb35fd9e37005ea356c7af"
Jul 16 22:48:53.822614 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:53.822597    2581 scope.go:117] "RemoveContainer" containerID="05496b8116e703d11d34a0ab39b433e10c01ae88e23f5dbdb29a01aed78b88b5"
Jul 16 22:48:54.598656 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:54.598627    2581 generic.go:356] "Generic (PLEG): container finished" podID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerID="d6470d35922983aa0ae42e811657a986d74d14ad880866c8ea91e2bd9c110ecd" exitCode=6
Jul 16 22:48:54.599083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:54.598704    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerDied","Data":"d6470d35922983aa0ae42e811657a986d74d14ad880866c8ea91e2bd9c110ecd"}
Jul 16 22:48:54.599083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:54.598773    2581 scope.go:117] "RemoveContainer" containerID="47d2af75f5ff8052f6ab05ddfb9d37a4f27c2c2bd1c8eb8e6a5d6d5323b4dcc8"
Jul 16 22:48:54.599083 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:54.599047    2581 scope.go:117] "RemoveContainer" containerID="d6470d35922983aa0ae42e811657a986d74d14ad880866c8ea91e2bd9c110ecd"
Jul 16 22:48:54.599290 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:48:54.599270    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-a6237863-r2hg9_odh-ai-gateway-infra(437ddb4b-0e45-4e57-a2f9-8357144a8d9d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d"
Jul 16 22:48:54.631148 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:54.631126    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:55.754884 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:55.754858    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:55.835537 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:55.835507    2581 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca\") pod \"437ddb4b-0e45-4e57-a2f9-8357144a8d9d\" (UID: \"437ddb4b-0e45-4e57-a2f9-8357144a8d9d\") "
Jul 16 22:48:55.835828 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:55.835806    2581 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "437ddb4b-0e45-4e57-a2f9-8357144a8d9d" (UID: "437ddb4b-0e45-4e57-a2f9-8357144a8d9d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:48:55.936048 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:55.935974    2581 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/437ddb4b-0e45-4e57-a2f9-8357144a8d9d-maas-api-service-ca\") on node \"ip-10-0-138-223.ec2.internal\" DevicePath \"\""
Jul 16 22:48:56.613955 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:56.613907    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9" event={"ID":"437ddb4b-0e45-4e57-a2f9-8357144a8d9d","Type":"ContainerDied","Data":"04e9b8adbd411249b5bc26f9f0997ee6066bb422d0d87ec222245158711d783b"}
Jul 16 22:48:56.613955 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:56.613944    2581 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"
Jul 16 22:48:56.614197 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:56.613977    2581 scope.go:117] "RemoveContainer" containerID="d6470d35922983aa0ae42e811657a986d74d14ad880866c8ea91e2bd9c110ecd"
Jul 16 22:48:56.634318 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:56.634294    2581 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:56.640803 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:56.640781    2581 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a6237863-r2hg9"]
Jul 16 22:48:57.091575 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:57.091547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zqkd4_fca4ff45-8365-4629-88a8-e67e2f1bfc7d/global-pull-secret-syncer/0.log"
Jul 16 22:48:57.160022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:57.159989    2581 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" path="/var/lib/kubelet/pods/437ddb4b-0e45-4e57-a2f9-8357144a8d9d/volumes"
Jul 16 22:48:57.224267 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:57.224240    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-ss5km_383e7ef0-501b-416f-ba30-712db4ed855b/konnectivity-agent/0.log"
Jul 16 22:48:57.278095 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:48:57.278053    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-223.ec2.internal_ec3998bf518cc797a77d2a84d9ceb141/haproxy/0.log"
Jul 16 22:49:03.261622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:03.261592    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-4dfh2_2cad4968-06e6-41c0-90f0-9443f79d4f44/manager/0.log"
Jul 16 22:49:03.533074 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:03.532986    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-87hdc_c25947ed-c099-4728-b170-10980f34bf79/limitador/0.log"
Jul 16 22:49:03.636906 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:03.636850    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-bhm6k_8a872677-006e-4af2-99cc-23f1ebaed274/manager/0.log"
Jul 16 22:49:05.570811 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.570783    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qppn2_4675264c-f22e-444a-b441-85f30adc3ef7/kube-state-metrics/0.log"
Jul 16 22:49:05.593787 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.593762    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qppn2_4675264c-f22e-444a-b441-85f30adc3ef7/kube-rbac-proxy-main/0.log"
Jul 16 22:49:05.615426 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.615397    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-qppn2_4675264c-f22e-444a-b441-85f30adc3ef7/kube-rbac-proxy-self/0.log"
Jul 16 22:49:05.866325 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.866293    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjq95_a558964d-b2aa-4cce-9733-aa8162b70c38/node-exporter/0.log"
Jul 16 22:49:05.887605 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.887578    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjq95_a558964d-b2aa-4cce-9733-aa8162b70c38/kube-rbac-proxy/0.log"
Jul 16 22:49:05.909031 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:05.909008    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-vjq95_a558964d-b2aa-4cce-9733-aa8162b70c38/init-textfile/0.log"
Jul 16 22:49:06.879460 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879425    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"]
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879719    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879732    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879742    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879750    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879762    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879770    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879782    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879789    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879804    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879812    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879828    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879836    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.879844 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879847    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879855    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879861    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879869    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879877    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879884    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.879991    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880005    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880015    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0c562dd1-ea43-47b8-95af-e20eb2907723" containerName="istio-proxy"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880023    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880031    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880038    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880045    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880053    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:06.880406 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.880063    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:06.884269 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.884249    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:06.887024 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.887003    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nx7gf\"/\"openshift-service-ca.crt\""
Jul 16 22:49:06.887751 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.887736    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nx7gf\"/\"default-dockercfg-pvwf8\""
Jul 16 22:49:06.887836 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.887751    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nx7gf\"/\"kube-root-ca.crt\""
Jul 16 22:49:06.895929 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.895911    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"]
Jul 16 22:49:06.932953 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.932934    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-proc\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:06.933054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.932959    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-podres\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:06.933054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.932990    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-lib-modules\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:06.933054 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.933041    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-sys\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:06.933157 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:06.933086    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8nms\" (UniqueName: \"kubernetes.io/projected/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-kube-api-access-b8nms\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.033975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.033945    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-proc\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.033975 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.033976    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-podres\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034007    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-lib-modules\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034054    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-sys\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034069    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-proc\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034122    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-sys\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034144 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034129    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-lib-modules\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034158    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8nms\" (UniqueName: \"kubernetes.io/projected/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-kube-api-access-b8nms\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.034330 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.034170    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-podres\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.042484 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.042462    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8nms\" (UniqueName: \"kubernetes.io/projected/eb81d52b-6f7a-4487-8d0e-e3f470f76a54-kube-api-access-b8nms\") pod \"perf-node-gather-daemonset-c7zft\" (UID: \"eb81d52b-6f7a-4487-8d0e-e3f470f76a54\") " pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.193820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.193748    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.341053 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.340945    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"]
Jul 16 22:49:07.343337 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:49:07.343309    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb81d52b_6f7a_4487_8d0e_e3f470f76a54.slice/crio-32161a5dd66680bf1f268c236cc252c6d814e420b81f6f451d30cc172522e1a0 WatchSource:0}: Error finding container 32161a5dd66680bf1f268c236cc252c6d814e420b81f6f451d30cc172522e1a0: Status 404 returned error can't find the container with id 32161a5dd66680bf1f268c236cc252c6d814e420b81f6f451d30cc172522e1a0
Jul 16 22:49:07.657612 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.657587    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-645r4_f7341c14-3ac0-4f15-8cd1-1f88bdf1717e/networking-console-plugin/0.log"
Jul 16 22:49:07.665622 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.665598    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft" event={"ID":"eb81d52b-6f7a-4487-8d0e-e3f470f76a54","Type":"ContainerStarted","Data":"fd19fcce5818c144c5dc47a92ce657e4eb0ab5ebef88bd23e92be2a76347aedc"}
Jul 16 22:49:07.665793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.665651    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft" event={"ID":"eb81d52b-6f7a-4487-8d0e-e3f470f76a54","Type":"ContainerStarted","Data":"32161a5dd66680bf1f268c236cc252c6d814e420b81f6f451d30cc172522e1a0"}
Jul 16 22:49:07.665793 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.665681    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:07.682440 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:07.682394    2581 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft" podStartSLOduration=1.6823807469999998 podStartE2EDuration="1.682380747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:49:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:49:07.681808736 +0000 UTC m=+3735.011248637" watchObservedRunningTime="2026-07-16 22:49:07.682380747 +0000 UTC m=+3735.011820648"
Jul 16 22:49:10.358565 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:10.358534    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fx9mf_12cd7b07-a130-4366-b420-03897d2ee326/dns/0.log"
Jul 16 22:49:10.382492 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:10.382466    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-fx9mf_12cd7b07-a130-4366-b420-03897d2ee326/kube-rbac-proxy/0.log"
Jul 16 22:49:10.562576 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:10.562547    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gq6wg_0cc7a5d8-eda1-4e1d-9421-a8933d99c709/dns-node-resolver/0.log"
Jul 16 22:49:11.152281 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:11.152244    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5cbdf84548-xlnd9_a50e695f-fbcf-4c8e-9dfa-a56259f9abe6/registry/0.log"
Jul 16 22:49:11.250889 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:11.250856    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-q7lj4_a599a220-6fa7-4d9e-be51-52258ab5f6c0/node-ca/0.log"
Jul 16 22:49:12.323717 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:12.323685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4m9wss_083b09d1-de49-4950-a8fa-75b4cc05a71d/istio-proxy/0.log"
Jul 16 22:49:12.540498 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:12.540468    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnq2_d287d78c-ded5-4b18-896c-c64ab048bcd3/discovery/0.log"
Jul 16 22:49:12.653169 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:12.653143    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-xhws9_30709861-ba73-4a0e-8306-5f3e1b991dee/kube-auth-proxy/0.log"
Jul 16 22:49:13.574328 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:13.574298    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-7prpt_9fe2c42b-98c5-491d-b75d-a817769e7c85/serve-healthcheck-canary/0.log"
Jul 16 22:49:13.680462 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:13.680434    2581 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nx7gf/perf-node-gather-daemonset-c7zft"
Jul 16 22:49:14.147991 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:14.147960    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2455k_6bc30111-495d-430a-9792-1a3021fd6c78/kube-rbac-proxy/0.log"
Jul 16 22:49:14.173090 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:14.173067    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2455k_6bc30111-495d-430a-9792-1a3021fd6c78/exporter/0.log"
Jul 16 22:49:14.195720 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:14.195685    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2455k_6bc30111-495d-430a-9792-1a3021fd6c78/extractor/0.log"
Jul 16 22:49:16.785413 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:16.785377    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-cgdjh_2f807aa4-2b3c-42c9-bb11-0c42de8ca13c/manager/0.log"
Jul 16 22:49:17.028088 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.027995    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-z2hnh_bad5efa6-6e90-43d3-838a-d553b568c204/manager/0.log"
Jul 16 22:49:17.182928 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.182897    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4m9wss_083b09d1-de49-4950-a8fa-75b4cc05a71d/istio-proxy/0.log"
Jul 16 22:49:17.201820 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.201795    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-fgrdt_fcfd39c5-1e47-4ecc-a2bb-7c6bcaf921c2/server/0.log"
Jul 16 22:49:17.309341 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.309259    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-pzzpd_a852da53-3657-47a1-b8f0-78f8dfd0985a/manager/0.log"
Jul 16 22:49:17.344355 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.344321    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-df6666d-pzzpd_a852da53-3657-47a1-b8f0-78f8dfd0985a/copy-manifests/0.log"
Jul 16 22:49:17.376850 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.376826    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnq2_d287d78c-ded5-4b18-896c-c64ab048bcd3/discovery/0.log"
Jul 16 22:49:17.410022 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:17.405968    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-xhws9_30709861-ba73-4a0e-8306-5f3e1b991dee/kube-auth-proxy/0.log"
Jul 16 22:49:18.097171 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097139    2581 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"]
Jul 16 22:49:18.097555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097499    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:18.097555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097510    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="da840848-57cf-47e0-9327-5df27b15e8ff" containerName="revoke-keys"
Jul 16 22:49:18.097555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097523    2581 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:18.097555 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097528    2581 state_mem.go:107] "Deleted CPUSet assignment" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:18.097701 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097580    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="437ddb4b-0e45-4e57-a2f9-8357144a8d9d" containerName="revoke-keys"
Jul 16 22:49:18.097701 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.097590    2581 memory_manager.go:368] "RemoveStaleState removing state" podUID="0cf5b5c1-d490-438f-a51b-9c3f762e5fc3" containerName="revoke-keys"
Jul 16 22:49:18.100648 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.100632    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"
Jul 16 22:49:18.103455 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.103434    2581 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:49:18.103624 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.103455    2581 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-h7ffk\""
Jul 16 22:49:18.108713 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.108688    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"]
Jul 16 22:49:18.221131 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.221093    2581 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd60e7a-6156-4179-abef-1244a351aeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph\" (UID: \"4cd60e7a-6156-4179-abef-1244a351aeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"
Jul 16 22:49:18.322348 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.322276    2581 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd60e7a-6156-4179-abef-1244a351aeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph\" (UID: \"4cd60e7a-6156-4179-abef-1244a351aeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"
Jul 16 22:49:18.322809 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.322792    2581 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4cd60e7a-6156-4179-abef-1244a351aeaa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph\" (UID: \"4cd60e7a-6156-4179-abef-1244a351aeaa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"
Jul 16 22:49:18.412383 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.412353    2581 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"
Jul 16 22:49:18.544192 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.544171    2581 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph"]
Jul 16 22:49:18.546432 ip-10-0-138-223 kubenswrapper[2581]: W0716 22:49:18.546387    2581 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cd60e7a_6156_4179_abef_1244a351aeaa.slice/crio-71f561e4d39f4b5024434d6e420e3000cad27eadc2cd4fdd9dd2b1cf7f578431 WatchSource:0}: Error finding container 71f561e4d39f4b5024434d6e420e3000cad27eadc2cd4fdd9dd2b1cf7f578431: Status 404 returned error can't find the container with id 71f561e4d39f4b5024434d6e420e3000cad27eadc2cd4fdd9dd2b1cf7f578431
Jul 16 22:49:18.714881 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.714842    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph" event={"ID":"4cd60e7a-6156-4179-abef-1244a351aeaa","Type":"ContainerStarted","Data":"71f561e4d39f4b5024434d6e420e3000cad27eadc2cd4fdd9dd2b1cf7f578431"}
Jul 16 22:49:18.854253 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:18.854210    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-6576859cf4m9wss_083b09d1-de49-4950-a8fa-75b4cc05a71d/istio-proxy/0.log"
Jul 16 22:49:19.056651 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:19.056621    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-dxnq2_d287d78c-ded5-4b18-896c-c64ab048bcd3/discovery/0.log"
Jul 16 22:49:19.089764 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:19.089737    2581 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5c8bcbdbf8-xhws9_30709861-ba73-4a0e-8306-5f3e1b991dee/kube-auth-proxy/0.log"
Jul 16 22:49:19.720421 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:19.720383    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cd60e7a-6156-4179-abef-1244a351aeaa" containerID="efb596a62656df67961e62af512df0ea70bdc78cbbc900eb89f9dd71fee1fa73" exitCode=6
Jul 16 22:49:19.720818 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:19.720481    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph" event={"ID":"4cd60e7a-6156-4179-abef-1244a351aeaa","Type":"ContainerDied","Data":"efb596a62656df67961e62af512df0ea70bdc78cbbc900eb89f9dd71fee1fa73"}
Jul 16 22:49:19.720818 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:19.720629    2581 scope.go:117] "RemoveContainer" containerID="efb596a62656df67961e62af512df0ea70bdc78cbbc900eb89f9dd71fee1fa73"
Jul 16 22:49:20.727099 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:20.727017    2581 generic.go:356] "Generic (PLEG): container finished" podID="4cd60e7a-6156-4179-abef-1244a351aeaa" containerID="a08247ed7fbe96f72d8c2f11e77edc912fd50e931bcb21e2a08f45f616c058fa" exitCode=6
Jul 16 22:49:20.727515 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:20.727104    2581 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph" event={"ID":"4cd60e7a-6156-4179-abef-1244a351aeaa","Type":"ContainerDied","Data":"a08247ed7fbe96f72d8c2f11e77edc912fd50e931bcb21e2a08f45f616c058fa"}
Jul 16 22:49:20.727515 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:20.727165    2581 scope.go:117] "RemoveContainer" containerID="efb596a62656df67961e62af512df0ea70bdc78cbbc900eb89f9dd71fee1fa73"
Jul 16 22:49:20.727515 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:20.727262    2581 scope.go:117] "RemoveContainer" containerID="a08247ed7fbe96f72d8c2f11e77edc912fd50e931bcb21e2a08f45f616c058fa"
Jul 16 22:49:20.727515 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:49:20.727484    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph_odh-ai-gateway-infra(4cd60e7a-6156-4179-abef-1244a351aeaa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph" podUID="4cd60e7a-6156-4179-abef-1244a351aeaa"
Jul 16 22:49:21.733769 ip-10-0-138-223 kubenswrapper[2581]: I0716 22:49:21.733738    2581 scope.go:117] "RemoveContainer" containerID="a08247ed7fbe96f72d8c2f11e77edc912fd50e931bcb21e2a08f45f616c058fa"
Jul 16 22:49:21.734257 ip-10-0-138-223 kubenswrapper[2581]: E0716 22:49:21.733977    2581 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph_odh-ai-gateway-infra(4cd60e7a-6156-4179-abef-1244a351aeaa)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-af8fc2dc-ll4ph" podUID="4cd60e7a-6156-4179-abef-1244a351aeaa"