Jul 16 21:49:03.516302 ip-10-0-131-79 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:49:03.918180 ip-10-0-131-79 kubenswrapper[2555]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:03.918180 ip-10-0-131-79 kubenswrapper[2555]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:49:03.918180 ip-10-0-131-79 kubenswrapper[2555]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:03.918180 ip-10-0-131-79 kubenswrapper[2555]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:49:03.918180 ip-10-0-131-79 kubenswrapper[2555]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:49:03.920034 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.919936    2555 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:49:03.923968 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923953    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.923968 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923968    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923972    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923975    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923978    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923981    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923983    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923986    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923988    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923991    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923995    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.923997    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924000    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924002    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924025    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924028    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924031    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924035    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924039    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924042    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924045    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.924058 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924048    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924051    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924053    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924056    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924059    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924062    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924065    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924067    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924070    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924073    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924076    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924078    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924082    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924084    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924087    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924089    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924092    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924094    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924096    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924099    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.924749 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924101    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924103    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924106    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924108    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924110    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924113    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924116    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924118    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924120    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924123    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924125    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924127    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924130    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924132    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924135    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924138    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924140    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924143    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924145    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.925557 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924149    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924152    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924155    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924158    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924160    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924163    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924165    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924167    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924170    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924172    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924174    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924176    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924179    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924181    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924183    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924260    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924270    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924275    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924281    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.926262 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924285    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924301    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924304    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924307    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924310    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924313    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924315    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924318    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924327    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924333    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924338    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924342    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924346    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.924351    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926128    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926141    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926145    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926148    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926151    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.926754 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926154    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926862    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926869    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926872    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926876    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926881    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926884    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926886    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926889    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926891    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926894    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926897    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926899    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926902    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926904    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926906    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926909    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926911    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926913    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.927274 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926916    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926919    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926922    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926924    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926927    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926929    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926931    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926934    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926936    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926938    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926941    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926943    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926945    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926948    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926950    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926953    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926956    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926958    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926962    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926964    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926966    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.927782 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926969    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926971    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926973    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926975    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926978    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926980    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926982    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926985    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926987    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926990    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926992    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926994    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.926998    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927000    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927002    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927005    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927027    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927030    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927032    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927035    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.928351 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927037    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927040    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927045    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927048    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927051    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927054    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927057    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927060    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927062    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927065    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927068    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927070    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927073    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927076    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927079    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927081    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927084    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927087    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927089    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927092    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.928881 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927094    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927098    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927102    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927105    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927108    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927111    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927114    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927117    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927120    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927122    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927126    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927128    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927131    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927133    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927136    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927138    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927141    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927143    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927147    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.929422 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927149    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927539    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927544    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927547    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927550    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927553    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927555    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927558    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927560    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927562    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927565    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927568    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927570    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927572    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927575    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927577    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927580    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927583    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927585    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927587    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.929933 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927590    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927592    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927595    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927597    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927600    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927611    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927615    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927617    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927620    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927622    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927624    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927627    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927630    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927633    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927635    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927638    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927640    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927643    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927645    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927647    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.930454 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927650    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927652    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927654    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927657    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927659    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927662    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927664    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927668    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927671    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927674    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927677    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927679    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927681    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927684    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927687    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927689    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927692    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927694    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927697    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927700    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.930975 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927702    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927705    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927707    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927710    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927712    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927716    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927718    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927720    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927723    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927726    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927728    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927730    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927733    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927735    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927737    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927740    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927743    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927745    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927747    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927750    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.931552 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927752    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927754    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927756    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927759    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927761    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927764    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927766    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927769    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927771    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927774    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927778    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927781    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927785    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927788    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927791    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927795    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927798    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927801    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927803    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.932112 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.927806    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927886    2555 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927892    2555 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927898    2555 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927902    2555 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927906    2555 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927909    2555 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927913    2555 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927917    2555 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927920    2555 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927923    2555 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927927    2555 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927930    2555 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927932    2555 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927935    2555 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927938    2555 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927941    2555 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927944    2555 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927947    2555 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927953    2555 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927956    2555 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927959    2555 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927961    2555 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927965    2555 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:49:03.932603 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927968    2555 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927971    2555 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927974    2555 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927977    2555 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927980    2555 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927983    2555 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927986    2555 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927989    2555 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927992    2555 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927995    2555 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.927998    2555 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928001    2555 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928003    2555 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928026    2555 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928029    2555 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928033    2555 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928036    2555 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928039    2555 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928042    2555 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928044    2555 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928048    2555 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928051    2555 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928054    2555 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928057    2555 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928059    2555 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:49:03.933258 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928065    2555 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928068    2555 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928071    2555 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928073    2555 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928076    2555 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928079    2555 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928082    2555 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928085    2555 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928088    2555 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928091    2555 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928095    2555 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928098    2555 flags.go:64] FLAG: --help="false"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928100    2555 flags.go:64] FLAG: --hostname-override="ip-10-0-131-79.ec2.internal"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928103    2555 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928106    2555 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928109    2555 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928112    2555 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928115    2555 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928118    2555 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928120    2555 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928123    2555 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928125    2555 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928128    2555 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928131    2555 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:49:03.933913 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928134    2555 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928137    2555 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928139    2555 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928142    2555 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928145    2555 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928147    2555 flags.go:64] FLAG: --lock-file=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928150    2555 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928153    2555 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928157    2555 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928162    2555 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928165    2555 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928167    2555 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928170    2555 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928173    2555 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928176    2555 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928178    2555 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928181    2555 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928185    2555 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928188    2555 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928192    2555 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928195    2555 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928198    2555 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928200    2555 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928203    2555 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928206    2555 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:49:03.934564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928208    2555 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928211    2555 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928218    2555 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928221    2555 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928224    2555 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928227    2555 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928230    2555 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928234    2555 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928237    2555 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928240    2555 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928243    2555 flags.go:64] FLAG: --port="10250"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928246    2555 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928248    2555 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0b24ec591567546fb"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928251    2555 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928254    2555 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928257    2555 flags.go:64] FLAG: --register-node="true"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928261    2555 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928264    2555 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928267    2555 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928270    2555 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928273    2555 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928276    2555 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928279    2555 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928282    2555 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928284    2555 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:49:03.935761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928287    2555 flags.go:64] FLAG: --runonce="false"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928290    2555 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928293    2555 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928296    2555 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928299    2555 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928302    2555 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928304    2555 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928307    2555 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928311    2555 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928313    2555 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928316    2555 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928319    2555 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928321    2555 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928324    2555 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928327    2555 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928329    2555 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928334    2555 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928337    2555 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928339    2555 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928344    2555 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928347    2555 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928349    2555 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928354    2555 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928358    2555 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928361    2555 flags.go:64] FLAG: --v="2"
Jul 16 21:49:03.936867 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928370    2555 flags.go:64] FLAG: --version="false"
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928374    2555 flags.go:64] FLAG: --vmodule=""
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928377    2555 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.928380    2555 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928494    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928498    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928501    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928503    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928506    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928509    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928512    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928514    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928516    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928520    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928524    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928527    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928529    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928532    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928535    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.938159 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928537    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928540    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928543    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928545    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928547    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928550    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928552    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928555    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928557    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928560    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928564    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928568    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928570    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928573    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928575    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928577    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928580    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928583    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928585    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928588    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.938763 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928590    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928593    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928595    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928597    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928600    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928603    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928605    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928608    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928610    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928612    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928615    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928618    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928622    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928625    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928628    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928630    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928633    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928635    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928637    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928640    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.939642 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928642    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928645    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928648    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928652    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928655    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928657    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928659    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928662    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928665    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928669    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928671    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928674    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928676    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928679    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928681    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928684    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928686    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928689    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928691    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.940441 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928694    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928697    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928699    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928702    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928704    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928706    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928709    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928711    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928713    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928716    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928718    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928721    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928723    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928725    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928728    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928731    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928736    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928738    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928740    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928743    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.941037 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928745    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928748    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928750    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928752    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.928755    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.929393    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.936884    2555 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.936905    2555 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.936931    2555 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.936940    2555 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937044    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937052    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937058    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937062    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937066    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.941610 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937071    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937075    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937079    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937083    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937088    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937092    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937096    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937100    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937104    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937108    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937113    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937117    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937122    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937126    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937130    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937134    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937139    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937142    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937147    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937151    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.942090 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937155    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937160    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937164    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937168    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937177    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937183    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937187    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937192    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937197    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937201    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937205    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937210    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937215    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937219    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937223    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937227    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937231    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937235    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937240    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.942656 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937244    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937249    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937253    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937257    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937261    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937265    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937269    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937273    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937277    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937282    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937286    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937290    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937294    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937298    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937302    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937308    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937314    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937319    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937325    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.943176 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937330    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937334    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937339    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937343    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937348    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937353    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937357    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937361    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937365    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937369    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937373    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937378    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937382    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937386    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937390    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937394    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937398    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937403    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937407    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937410    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.943730 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937415    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937419    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937423    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937427    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937433    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937438    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937443    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937447    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937451    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937455    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937460    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937465    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937470    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937474    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937478    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.944272 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937482    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.937490    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937675    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937683    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937688    2555 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937693    2555 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937698    2555 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937702    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937707    2555 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937711    2555 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937715    2555 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937720    2555 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937724    2555 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937728    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937732    2555 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:49:03.944678 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937736    2555 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937740    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937744    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937749    2555 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937753    2555 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937756    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937761    2555 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937765    2555 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937769    2555 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937773    2555 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937777    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937781    2555 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937786    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937791    2555 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937795    2555 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937799    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937803    2555 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937808    2555 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937812    2555 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937816    2555 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:49:03.945098 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937820    2555 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937824    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937828    2555 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937832    2555 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937836    2555 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937842    2555 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937847    2555 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937853    2555 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937857    2555 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937862    2555 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937866    2555 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937871    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937875    2555 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937881    2555 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937886    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937891    2555 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937895    2555 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937899    2555 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937903    2555 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:49:03.945777 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937907    2555 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937911    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937916    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937920    2555 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937924    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937928    2555 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937932    2555 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937936    2555 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937940    2555 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937946    2555 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937950    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937954    2555 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937958    2555 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937962    2555 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937966    2555 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937970    2555 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937974    2555 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937978    2555 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937982    2555 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937986    2555 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:49:03.946703 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937990    2555 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937994    2555 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.937997    2555 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938001    2555 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938026    2555 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938031    2555 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938034    2555 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938038    2555 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938042    2555 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938045    2555 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938050    2555 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938053    2555 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938057    2555 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938060    2555 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938064    2555 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938068    2555 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938071    2555 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938075    2555 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938079    2555 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:49:03.947376 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938083    2555 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938088    2555 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938093    2555 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938099    2555 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938103    2555 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938108    2555 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938113    2555 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:03.938117    2555 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.938127    2555 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.938274    2555 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.940891    2555 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.942510    2555 server.go:1019] "Starting client certificate rotation"
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.942616    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:49:03.947873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.942677    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:49:03.948261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.946069    2555 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:49:03.949877 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.949856    2555 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:49:03.961065 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.961048    2555 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:49:03.963896 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.963878    2555 log.go:25] "Validated CRI v1 image API"
Jul 16 21:49:03.965385 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.965368    2555 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:49:03.967902 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.967886    2555 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 9e70cbe9-512b-4b00-b02f-c0a47f104f72:/dev/nvme0n1p3 e9b9fb0b-13cb-45c3-b91e-f5cdd37c5b67:/dev/nvme0n1p4]
Jul 16 21:49:03.967968 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.967902    2555 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:49:03.974120 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.973994    2555 manager.go:216] Machine: {Timestamp:2026-07-16 21:49:03.971812605 +0000 UTC m=+0.333319092 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3135015 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2b720326dcb75cc769035deec849a2 SystemUUID:ec2b7203-26dc-b75c-c769-035deec849a2 BootID:064e28e1-a2ac-437e-aa4b-eb6a605eadcd Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:00:fd:ca:4a:2d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:00:fd:ca:4a:2d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:f6:e0:09:57:73:4e Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:49:03.974120 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.974116    2555 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:49:03.974234 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.974197    2555 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:49:03.977311 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.977278    2555 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:49:03.977447 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.977313    2555 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-79.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:49:03.978069 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.978058    2555 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:49:03.978069 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.978070    2555 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:49:03.978142 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.978078    2555 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:49:03.978717 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.978708    2555 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:49:03.980030 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.980004    2555 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:49:03.981749 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.981733    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:49:03.983304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983293    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:49:03.983413 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983404    2555 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:49:03.983453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983432    2555 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:49:03.983453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983442    2555 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:49:03.983453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983452    2555 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:49:03.983564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983460    2555 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:49:03.983564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983469    2555 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:49:03.983825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983803    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:49:03.983825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.983822    2555 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:49:03.985891 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.985875    2555 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:49:03.987898 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.987882    2555 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:49:03.987956 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.987924    2555 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:49:03.989122 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989111    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989130    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989137    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989143    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989153    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989159    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:49:03.989166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989168    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989174    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989181    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989188    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989197    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989204    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989228    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:49:03.989376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.989234    2555 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:49:03.992886 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.992873    2555 server.go:1268] "Started kubelet"
Jul 16 21:49:03.992998 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.992954    2555 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:49:03.993081 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.993040    2555 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:49:03.993129 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.992977    2555 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:49:03.993765 ip-10-0-131-79 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:49:03.995253 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.995235    2555 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:49:03.995592 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.995579    2555 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:49:03.998942 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:03.998918    2555 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-79.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:49:03.999290 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:03.999264    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:49:03.999290 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:03.999270    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:49:03.999634 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:03.999617    2555 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:49:04.002989 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:03.999252    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-79.ec2.internal.18c2e380a140eac2  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-79.ec2.internal,UID:ip-10-0-131-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-79.ec2.internal,},FirstTimestamp:2026-07-16 21:49:03.992842946 +0000 UTC m=+0.354349433,LastTimestamp:2026-07-16 21:49:03.992842946 +0000 UTC m=+0.354349433,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-79.ec2.internal,}"
Jul 16 21:49:04.003255 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003240    2555 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:49:04.003369 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003347    2555 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:49:04.003435 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.003409    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.003846 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003748    2555 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:49:04.003846 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003750    2555 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:49:04.003846 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003776    2555 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:49:04.004057 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003943    2555 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:49:04.004057 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.003952    2555 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:49:04.004213 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004197    2555 factory.go:55] Registering systemd factory
Jul 16 21:49:04.004305 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004214    2555 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:49:04.004393 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004379    2555 factory.go:153] Registering CRI-O factory
Jul 16 21:49:04.004450 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004398    2555 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:49:04.004501 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004450    2555 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:49:04.004501 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004474    2555 factory.go:103] Registering Raw factory
Jul 16 21:49:04.004501 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004490    2555 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:49:04.004855 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.004841    2555 manager.go:349] Starting recovery of all containers
Jul 16 21:49:04.006026 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.005988    2555 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-79.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 16 21:49:04.006776 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.006753    2555 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 16 21:49:04.015802 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.015784    2555 manager.go:354] Recovery completed
Jul 16 21:49:04.020966 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.020955    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.023341 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.022819    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.023341 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.022850    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.023341 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.022880    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.024411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.024392    2555 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:49:04.024411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.024408    2555 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:49:04.024509 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.024415    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.024509 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.024423    2555 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:49:04.025178 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025162    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.025260 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025195    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.025260 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025210    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.025639 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025618    2555 policy_none.go:49] "None policy: Start"
Jul 16 21:49:04.025639 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025639    2555 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:49:04.025807 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.025650    2555 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:49:04.026784 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.026772    2555 policy_none.go:47] "Start"
Jul 16 21:49:04.040812 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.040745    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-79.ec2.internal.18c2e380a30a9b06  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-79.ec2.internal,UID:ip-10-0-131-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-79.ec2.internal,},FirstTimestamp:2026-07-16 21:49:04.022838022 +0000 UTC m=+0.384344510,LastTimestamp:2026-07-16 21:49:04.022838022 +0000 UTC m=+0.384344510,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-79.ec2.internal,}"
Jul 16 21:49:04.060805 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.060739    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-79.ec2.internal.18c2e380a30adf27  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-79.ec2.internal,UID:ip-10-0-131-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-131-79.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-131-79.ec2.internal,},FirstTimestamp:2026-07-16 21:49:04.022855463 +0000 UTC m=+0.384361954,LastTimestamp:2026-07-16 21:49:04.022855463 +0000 UTC m=+0.384361954,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-79.ec2.internal,}"
Jul 16 21:49:04.062563 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.062550    2555 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:49:04.062627 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.062578    2555 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:49:04.062627 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.062587    2555 server.go:87] "Starting device plugin registration server"
Jul 16 21:49:04.062809 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.062793    2555 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:49:04.062908 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.062808    2555 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:49:04.062908 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.062897    2555 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:49:04.063047 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.063001    2555 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:49:04.063047 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.063029    2555 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:49:04.063521 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.063501    2555 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:49:04.063600 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.063556    2555 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.070466 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.070452    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwdvf"
Jul 16 21:49:04.080407 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.080340    2555 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-79.ec2.internal.18c2e380a30b594f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-79.ec2.internal,UID:ip-10-0-131-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-131-79.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-131-79.ec2.internal,},FirstTimestamp:2026-07-16 21:49:04.022886735 +0000 UTC m=+0.384393225,LastTimestamp:2026-07-16 21:49:04.022886735 +0000 UTC m=+0.384393225,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-79.ec2.internal,}"
Jul 16 21:49:04.092575 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.092496    2555 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-131-79.ec2.internal.18c2e380a30a9b06\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-79.ec2.internal.18c2e380a30a9b06  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-79.ec2.internal,UID:ip-10-0-131-79.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-79.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-79.ec2.internal,},FirstTimestamp:2026-07-16 21:49:04.022838022 +0000 UTC m=+0.384344510,LastTimestamp:2026-07-16 21:49:04.025177888 +0000 UTC m=+0.386684380,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-79.ec2.internal,}"
Jul 16 21:49:04.097398 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.097373    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-fwdvf"
Jul 16 21:49:04.143962 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.143923    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:49:04.145280 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.145261    2555 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:49:04.145396 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.145299    2555 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:49:04.145396 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.145319    2555 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:49:04.145396 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.145350    2555 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:49:04.149467 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.149448    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:04.163771 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.163746    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.165121 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.165105    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.165227 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.165139    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.165227 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.165154    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.165227 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.165184    2555 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.185453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.185383    2555 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.185453 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.185411    2555 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-79.ec2.internal\": node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.225314 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.225284    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.246114 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.246071    2555 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"]
Jul 16 21:49:04.246192 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.246162    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.247161 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.247134    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.247277 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.247168    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.247277 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.247183    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.248822 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.248807    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.248921 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.248903    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.248978 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.248939    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.249508 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249489    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.249608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249517    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.249608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249530    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.249608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249554    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.249608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249577    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.249608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.249591    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.250697 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.250684    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.250748 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.250708    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:04.251383 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.251362    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:04.251464 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.251392    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:04.251464 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.251406    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:04.279973 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.279949    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-79.ec2.internal\" not found" node="ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.284809 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.284792    2555 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-79.ec2.internal\" not found" node="ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.305059 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.305028    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.305059 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.305061    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.305239 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.305079    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8d0181de62365719ca8addca6e9fe43-config\") pod \"kube-apiserver-proxy-ip-10-0-131-79.ec2.internal\" (UID: \"e8d0181de62365719ca8addca6e9fe43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.326088 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.326056    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.405465 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405432    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.405624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405513    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.405624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405517    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.405624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405533    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/0dba07077d86471426887abd01210c23-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal\" (UID: \"0dba07077d86471426887abd01210c23\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.405624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405566    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8d0181de62365719ca8addca6e9fe43-config\") pod \"kube-apiserver-proxy-ip-10-0-131-79.ec2.internal\" (UID: \"e8d0181de62365719ca8addca6e9fe43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.405624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.405582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e8d0181de62365719ca8addca6e9fe43-config\") pod \"kube-apiserver-proxy-ip-10-0-131-79.ec2.internal\" (UID: \"e8d0181de62365719ca8addca6e9fe43\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.426871 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.426844    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.527415 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.527344    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.584064 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.584027    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.588063 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:04.587832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:04.628173 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.628144    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.729172 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.729133    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:04.829646 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:04.829570    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.454547 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.454515    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.466694 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.466671    2555 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:49:06.466832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.466812    2555 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:49:06.554750 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.554715    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.571593 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.571567    2555 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:06.627388 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.627328    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:44:05 +0000 UTC" deadline="2028-02-07 07:11:07.681960015 +0000 UTC"
Jul 16 21:49:06.627388 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.627387    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13689h22m1.054577324s"
Jul 16 21:49:06.627578 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.627401    2555 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:06.632129 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:06.632043    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0dba07077d86471426887abd01210c23.slice/crio-d8a98ff06ac4e34b3782fc440ac67e111fb3c4dc2c45b3a8c72c783f5db731c9 WatchSource:0}: Error finding container d8a98ff06ac4e34b3782fc440ac67e111fb3c4dc2c45b3a8c72c783f5db731c9: Status 404 returned error can't find the container with id d8a98ff06ac4e34b3782fc440ac67e111fb3c4dc2c45b3a8c72c783f5db731c9
Jul 16 21:49:06.632446 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:06.632423    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8d0181de62365719ca8addca6e9fe43.slice/crio-2040b9a9c11183b9b624800923f960737d3da6a838777b06a7b69908762d7163 WatchSource:0}: Error finding container 2040b9a9c11183b9b624800923f960737d3da6a838777b06a7b69908762d7163: Status 404 returned error can't find the container with id 2040b9a9c11183b9b624800923f960737d3da6a838777b06a7b69908762d7163
Jul 16 21:49:06.636529 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.636516    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:49:06.654834 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.654809    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.655892 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.655881    2555 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:49:06.656950 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.656934    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:49:06.657037 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.656963    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:49:06.657037 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.656973    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:49:06.673753 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.673704    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal" event={"ID":"e8d0181de62365719ca8addca6e9fe43","Type":"ContainerStarted","Data":"2040b9a9c11183b9b624800923f960737d3da6a838777b06a7b69908762d7163"}
Jul 16 21:49:06.674843 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:06.674817    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal" event={"ID":"0dba07077d86471426887abd01210c23","Type":"ContainerStarted","Data":"d8a98ff06ac4e34b3782fc440ac67e111fb3c4dc2c45b3a8c72c783f5db731c9"}
Jul 16 21:49:06.755425 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.755398    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.855920 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.855890    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:06.956435 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:06.956354    2555 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-79.ec2.internal\" not found"
Jul 16 21:49:07.038503 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.038452    2555 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:07.128825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.128788    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:07.144453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.144424    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:49:07.145678 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.145654    2555 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal"
Jul 16 21:49:07.157365 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.157339    2555 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:49:07.510048 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.510016    2555 apiserver.go:52] "Watching apiserver"
Jul 16 21:49:07.520696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.520670    2555 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:49:07.522578 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.522548    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-8jflr","openshift-network-diagnostics/network-check-target-c9l6v","openshift-network-operator/iptables-alerter-wckd4","openshift-ovn-kubernetes/ovnkube-node-c9cwb","kube-system/konnectivity-agent-pvxqn","kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr","openshift-cluster-node-tuning-operator/tuned-wrpqt","openshift-dns/node-resolver-gbvhz","openshift-image-registry/node-ca-d2cf5","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal","openshift-multus/multus-6h5b5","kube-system/global-pull-secret-syncer-jwnj9","openshift-multus/multus-additional-cni-plugins-fh6tz"]
Jul 16 21:49:07.524103 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.524081    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.526721 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.526702    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.527440 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.527422    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-6ppmw\""
Jul 16 21:49:07.528041 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.528024    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.528196 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.528180    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.528490 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.528454    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.528914 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.528889    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.529135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.529118    2555 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:49:07.531029 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.531010    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.532798 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.532781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.532878 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.532843    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:07.533631 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.533613    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:49:07.534882 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.534811    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.536233 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.536213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.537632 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.537613    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.538991 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.538963    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.539607 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539589    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-var-lib-kubelet\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.539711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539620    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-netns\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.539711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-daemon-config\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.539711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539692    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-multus-certs\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539714    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cnibin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-systemd\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539758    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-etc-tuned\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539785    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-tmp\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539829    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-etc-kubernetes\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.539875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539857    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqb5m\" (UniqueName: \"kubernetes.io/projected/55de2ac7-3161-4f49-a52c-37b655f5c8b7-kube-api-access-mqb5m\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80d51c90-35f7-4d89-8ef7-ef6667c84f03-tmp-dir\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539905    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-conf\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-sys\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539946    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.539970    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-kubernetes\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540009    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-run\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540041    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv66\" (UniqueName: \"kubernetes.io/projected/487b236d-030e-4541-8ac0-5199b92f6b16-kube-api-access-tsv66\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-os-release\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540101    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-os-release\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540183    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-kubelet\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540225    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-cnibin\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540252    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540298    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-system-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540339    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-binary-copy\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540391    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4p2p\" (UniqueName: \"kubernetes.io/projected/486dac09-187b-468c-bae0-9ed832b5dd91-kube-api-access-z4p2p\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540493    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-modprobe-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540515    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-lib-modules\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-host\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540596    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysconfig\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.540650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540645    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-system-cni-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cni-binary-copy\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540700    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-k8s-cni-cncf-io\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540740    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-bin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540765    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-multus\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540787    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-hostroot\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540817    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540859    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-socket-dir-parent\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-conf-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80d51c90-35f7-4d89-8ef7-ef6667c84f03-hosts-file\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.541425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.540907    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7c4\" (UniqueName: \"kubernetes.io/projected/80d51c90-35f7-4d89-8ef7-ef6667c84f03-kube-api-access-cz7c4\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.541978 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.541623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.541978 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.541652    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:49:07.541978 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.541686    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:07.542946 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.542930    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:07.543039 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.542992    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:07.545952 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.545931    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:49:07.545952 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.545949    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-tfjd5\""
Jul 16 21:49:07.546096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.545958    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:49:07.551960 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.551920    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.552184 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.552166    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-gt7pf\""
Jul 16 21:49:07.552548 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.552510    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.552681 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.552615    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.553019 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.552991    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:49:07.553210 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553189    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:49:07.553300 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553197    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:49:07.553300 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553245    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:49:07.553300 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553259    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:49:07.553436 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553323    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.554032 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553858    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.554032 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553913    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.554032 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.553955    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.554322 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554303    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-rqhrz\""
Jul 16 21:49:07.554322 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554319    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:49:07.554480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554398    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:49:07.554480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554432    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.554587 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554566    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-872gw\""
Jul 16 21:49:07.554644 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554601    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-sfk48\""
Jul 16 21:49:07.554644 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554603    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:49:07.554644 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554624    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.554777 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554702    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-ptkh8\""
Jul 16 21:49:07.554777 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554732    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bcb4d\""
Jul 16 21:49:07.554882 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554816    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-whkjc\""
Jul 16 21:49:07.554935 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554901    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:49:07.554935 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554923    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:49:07.555035 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.554953    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:49:07.566294 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.566278    2555 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:49:07.610353 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.610324    2555 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7857"
Jul 16 21:49:07.626078 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.626048    2555 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7857"
Jul 16 21:49:07.630102 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.630068    2555 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:49:07.633735 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.633712    2555 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:49:07.642070 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642046    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-tmp\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642079    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80d51c90-35f7-4d89-8ef7-ef6667c84f03-tmp-dir\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642113    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-systemd-units\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtts5\" (UniqueName: \"kubernetes.io/projected/7052dd71-d93b-44d0-bf9c-64005d26e75d-kube-api-access-jtts5\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642159    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-dbus\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642185    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.642215 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642207    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-kubelet\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642229    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovn-node-metrics-cert\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642255    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-kubernetes\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642279    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-run\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642303    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grhjw\" (UniqueName: \"kubernetes.io/projected/91d7f8a0-15de-4df5-874a-b639628d4064-kube-api-access-grhjw\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-slash\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642351    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-log-socket\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642375    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df0d312c-dd41-4069-bd73-e9fc466f8544-host\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642379    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-kubernetes\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-run\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-cnibin\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642436    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/80d51c90-35f7-4d89-8ef7-ef6667c84f03-tmp-dir\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642448    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-cnibin\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642456    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642438    2555 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642510    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-etc-selinux\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.642539 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642536    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e959648-65d9-45ea-ac05-3522b5cc0041-agent-certs\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-kubelet-config\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-system-cni-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642610    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-tuning-conf-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642621    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4p2p\" (UniqueName: \"kubernetes.io/projected/486dac09-187b-468c-bae0-9ed832b5dd91-kube-api-access-z4p2p\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642664    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642679    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-system-cni-dir\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-netns\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642739    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642746    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-conf-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642780    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-var-lib-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-conf-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-ovn\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642831    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df0d312c-dd41-4069-bd73-e9fc466f8544-serviceca\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642859    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642885    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-socket-dir-parent\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.643285 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642910    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7c4\" (UniqueName: \"kubernetes.io/projected/80d51c90-35f7-4d89-8ef7-ef6667c84f03-kube-api-access-cz7c4\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642935    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-sys-fs\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642962    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-socket-dir-parent\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642977    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vt95\" (UniqueName: \"kubernetes.io/projected/df0d312c-dd41-4069-bd73-e9fc466f8544-kube-api-access-6vt95\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.642991    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643003    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643050    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-netns\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643083    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-multus-certs\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643109    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643136    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-bin\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643112    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-netns\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-env-overrides\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643197    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cnibin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643213    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-systemd\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643227    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-etc-tuned\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643242    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-etc-kubernetes\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643257    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqb5m\" (UniqueName: \"kubernetes.io/projected/55de2ac7-3161-4f49-a52c-37b655f5c8b7-kube-api-access-mqb5m\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643271    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-systemd\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643272    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cnibin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643310    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-etc-kubernetes\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643377    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-multus-certs\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-conf\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643521    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-sys\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643545    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-socket-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643572    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-registration-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643591    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-sys\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643595    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-systemd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643643    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-node-log\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643652    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysctl-conf\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643700    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-netd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643736    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnfdj\" (UniqueName: \"kubernetes.io/projected/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-kube-api-access-xnfdj\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.644865 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643770    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv66\" (UniqueName: \"kubernetes.io/projected/487b236d-030e-4541-8ac0-5199b92f6b16-kube-api-access-tsv66\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643796    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-os-release\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643820    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-os-release\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643843    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-kubelet\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-etc-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643890    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-script-lib\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/486dac09-187b-468c-bae0-9ed832b5dd91-os-release\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643932    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-iptables-alerter-script\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643959    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-system-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643983    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e959648-65d9-45ea-ac05-3522b5cc0041-konnectivity-ca\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.643966    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-kubelet\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644079    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-os-release\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644092    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-system-cni-dir\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644111    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-k8s-cni-cncf-io\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644159    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-bin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644180    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-run-k8s-cni-cncf-io\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-multus\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.645673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644212    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-bin\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644218    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-device-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644236    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-host-var-lib-cni-multus\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644243    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-binary-copy\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644302    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-modprobe-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysconfig\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cni-binary-copy\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644370    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-hostroot\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80d51c90-35f7-4d89-8ef7-ef6667c84f03-hosts-file\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644420    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfjxq\" (UniqueName: \"kubernetes.io/projected/b5645581-2719-434f-802d-4721384c2acc-kube-api-access-nfjxq\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-lib-modules\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644484    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-host\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644512    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-config\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644568    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-host-slash\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644599    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-var-lib-kubelet\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-daemon-config\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644770    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-cni-binary-copy\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644780    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-modprobe-d\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644820    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-etc-sysconfig\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644842    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-host\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644857    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/486dac09-187b-468c-bae0-9ed832b5dd91-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.644947    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-var-lib-kubelet\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.645317    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-cni-binary-copy\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.645360    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55de2ac7-3161-4f49-a52c-37b655f5c8b7-multus-daemon-config\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.645371    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55de2ac7-3161-4f49-a52c-37b655f5c8b7-hostroot\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.645416    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/80d51c90-35f7-4d89-8ef7-ef6667c84f03-hosts-file\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.645488    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/487b236d-030e-4541-8ac0-5199b92f6b16-lib-modules\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.646428    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-etc-tuned\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.646940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.646480    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/487b236d-030e-4541-8ac0-5199b92f6b16-tmp\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.667791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.667724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv66\" (UniqueName: \"kubernetes.io/projected/487b236d-030e-4541-8ac0-5199b92f6b16-kube-api-access-tsv66\") pod \"tuned-wrpqt\" (UID: \"487b236d-030e-4541-8ac0-5199b92f6b16\") " pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.671981 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.671957    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqb5m\" (UniqueName: \"kubernetes.io/projected/55de2ac7-3161-4f49-a52c-37b655f5c8b7-kube-api-access-mqb5m\") pod \"multus-6h5b5\" (UID: \"55de2ac7-3161-4f49-a52c-37b655f5c8b7\") " pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.672948 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.672923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4p2p\" (UniqueName: \"kubernetes.io/projected/486dac09-187b-468c-bae0-9ed832b5dd91-kube-api-access-z4p2p\") pod \"multus-additional-cni-plugins-fh6tz\" (UID: \"486dac09-187b-468c-bae0-9ed832b5dd91\") " pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.673923 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.673900    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7c4\" (UniqueName: \"kubernetes.io/projected/80d51c90-35f7-4d89-8ef7-ef6667c84f03-kube-api-access-cz7c4\") pod \"node-resolver-gbvhz\" (UID: \"80d51c90-35f7-4d89-8ef7-ef6667c84f03\") " pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.746097 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-sys-fs\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.746097 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746100    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vt95\" (UniqueName: \"kubernetes.io/projected/df0d312c-dd41-4069-bd73-e9fc466f8544-kube-api-access-6vt95\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746151    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746199    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-sys-fs\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746202    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-bin\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-bin\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746266    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-env-overrides\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746305    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.746328    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746330    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.746420    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:08.246384751 +0000 UTC m=+3.083438549 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746441    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-socket-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746485    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-registration-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.746496    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746512    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-systemd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.746543    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:08.246530352 +0000 UTC m=+3.083584132 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746559    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-node-log\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746590    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-netd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746595    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnfdj\" (UniqueName: \"kubernetes.io/projected/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-kube-api-access-xnfdj\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746650    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-etc-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746643    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.746696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-script-lib\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746700    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-iptables-alerter-script\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-etc-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e959648-65d9-45ea-ac05-3522b5cc0041-konnectivity-ca\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-node-log\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746779    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-device-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfjxq\" (UniqueName: \"kubernetes.io/projected/b5645581-2719-434f-802d-4721384c2acc-kube-api-access-nfjxq\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746815    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-env-overrides\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746841    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-config\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746860    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-registration-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746868    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746891    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-host-slash\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746924    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746950    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-systemd-units\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746973    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtts5\" (UniqueName: \"kubernetes.io/projected/7052dd71-d93b-44d0-bf9c-64005d26e75d-kube-api-access-jtts5\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746997    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-dbus\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747020    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-kubelet\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747043    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovn-node-metrics-cert\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grhjw\" (UniqueName: \"kubernetes.io/projected/91d7f8a0-15de-4df5-874a-b639628d4064-kube-api-access-grhjw\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747096    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-slash\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747118    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-log-socket\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747127    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/4e959648-65d9-45ea-ac05-3522b5cc0041-konnectivity-ca\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df0d312c-dd41-4069-bd73-e9fc466f8544-host\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-etc-selinux\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747189    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e959648-65d9-45ea-ac05-3522b5cc0041-agent-certs\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-kubelet-config\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747246    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-netns\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747274    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-var-lib-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-ovn\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747324    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df0d312c-dd41-4069-bd73-e9fc466f8544-serviceca\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-script-lib\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746561    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-systemd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-cni-netd\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747406    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-kubelet\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.747939 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747510    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-iptables-alerter-script\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747566    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-host-slash\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.746818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-socket-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747673    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-kubelet-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747707    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-systemd-units\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747863    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-kubelet-config\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-slash\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovnkube-config\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/df0d312c-dd41-4069-bd73-e9fc466f8544-host\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747978    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-var-lib-openvswitch\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-ovn-kubernetes\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.747990    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-run-ovn\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748015    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-host-run-netns\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748125    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-dbus\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748210    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-etc-selinux\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748222    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/91d7f8a0-15de-4df5-874a-b639628d4064-device-dir\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/df0d312c-dd41-4069-bd73-e9fc466f8544-serviceca\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.748590 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.748264    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/7052dd71-d93b-44d0-bf9c-64005d26e75d-log-socket\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.750554 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.750532    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/4e959648-65d9-45ea-ac05-3522b5cc0041-agent-certs\") pod \"konnectivity-agent-pvxqn\" (UID: \"4e959648-65d9-45ea-ac05-3522b5cc0041\") " pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.751018 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.751002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/7052dd71-d93b-44d0-bf9c-64005d26e75d-ovn-node-metrics-cert\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.762729 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.762674    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vt95\" (UniqueName: \"kubernetes.io/projected/df0d312c-dd41-4069-bd73-e9fc466f8544-kube-api-access-6vt95\") pod \"node-ca-d2cf5\" (UID: \"df0d312c-dd41-4069-bd73-e9fc466f8544\") " pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.770674 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.770650    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:07.770674 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.770677    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:07.770832 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.770688    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:07.770832 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:07.770750    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:08.270736682 +0000 UTC m=+3.107790460 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:07.773360 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.773330    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnfdj\" (UniqueName: \"kubernetes.io/projected/a2e4c212-c11d-4dec-b3db-76d86bf09ea5-kube-api-access-xnfdj\") pod \"iptables-alerter-wckd4\" (UID: \"a2e4c212-c11d-4dec-b3db-76d86bf09ea5\") " pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:07.775597 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.775577    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtts5\" (UniqueName: \"kubernetes.io/projected/7052dd71-d93b-44d0-bf9c-64005d26e75d-kube-api-access-jtts5\") pod \"ovnkube-node-c9cwb\" (UID: \"7052dd71-d93b-44d0-bf9c-64005d26e75d\") " pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.776303 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.776280    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grhjw\" (UniqueName: \"kubernetes.io/projected/91d7f8a0-15de-4df5-874a-b639628d4064-kube-api-access-grhjw\") pod \"aws-ebs-csi-driver-node-hzwhr\" (UID: \"91d7f8a0-15de-4df5-874a-b639628d4064\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.778173 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.778157    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfjxq\" (UniqueName: \"kubernetes.io/projected/b5645581-2719-434f-802d-4721384c2acc-kube-api-access-nfjxq\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:07.836620 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.836586    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wrpqt"
Jul 16 21:49:07.843694 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.843669    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-fh6tz"
Jul 16 21:49:07.854524 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.854496    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h5b5"
Jul 16 21:49:07.859709 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.859689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbvhz"
Jul 16 21:49:07.866711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.866689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr"
Jul 16 21:49:07.876201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.876178    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:07.882511 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.882492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:07.888152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.888128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-d2cf5"
Jul 16 21:49:07.894525 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:07.894503    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-wckd4"
Jul 16 21:49:08.251305 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.251224    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:08.251305 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.251260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:08.251531 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.251353    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:08.251531 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.251374    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:08.251531 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.251411    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:09.25139805 +0000 UTC m=+4.088451828 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:08.251531 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.251437    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:09.251419233 +0000 UTC m=+4.088473026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:08.352335 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.352299    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:08.352527 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.352478    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:08.352527 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.352510    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:08.352527 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.352524    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:08.352686 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.352593    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:09.352573027 +0000 UTC m=+4.189626821 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:08.541728 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.541701    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod487b236d_030e_4541_8ac0_5199b92f6b16.slice/crio-e450f3f31ae51a7993a2c8fcf5f157cedc0ac941a043c84d1f99f12f9d7db5fd WatchSource:0}: Error finding container e450f3f31ae51a7993a2c8fcf5f157cedc0ac941a043c84d1f99f12f9d7db5fd: Status 404 returned error can't find the container with id e450f3f31ae51a7993a2c8fcf5f157cedc0ac941a043c84d1f99f12f9d7db5fd
Jul 16 21:49:08.560066 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.560037    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod486dac09_187b_468c_bae0_9ed832b5dd91.slice/crio-aba30807b16e101eb9012575d3eebea9af4d0664bc72c9266945a8d7845b12ea WatchSource:0}: Error finding container aba30807b16e101eb9012575d3eebea9af4d0664bc72c9266945a8d7845b12ea: Status 404 returned error can't find the container with id aba30807b16e101eb9012575d3eebea9af4d0664bc72c9266945a8d7845b12ea
Jul 16 21:49:08.560855 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.560826    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e959648_65d9_45ea_ac05_3522b5cc0041.slice/crio-5e79a8cecddd1f11e3378e21bfdadda5b1b8f7e20d9f88ae117c235c1bb4c221 WatchSource:0}: Error finding container 5e79a8cecddd1f11e3378e21bfdadda5b1b8f7e20d9f88ae117c235c1bb4c221: Status 404 returned error can't find the container with id 5e79a8cecddd1f11e3378e21bfdadda5b1b8f7e20d9f88ae117c235c1bb4c221
Jul 16 21:49:08.561757 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.561739    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80d51c90_35f7_4d89_8ef7_ef6667c84f03.slice/crio-9e274183e9a4e4083851a6fb38c6ee3c75dc3892242b50d200e3b287fbd77c9d WatchSource:0}: Error finding container 9e274183e9a4e4083851a6fb38c6ee3c75dc3892242b50d200e3b287fbd77c9d: Status 404 returned error can't find the container with id 9e274183e9a4e4083851a6fb38c6ee3c75dc3892242b50d200e3b287fbd77c9d
Jul 16 21:49:08.562990 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.562967    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55de2ac7_3161_4f49_a52c_37b655f5c8b7.slice/crio-d8076816b63d8376b9ebda69cf6992bb7f592c10e0f039af2f0a87d149dd1eee WatchSource:0}: Error finding container d8076816b63d8376b9ebda69cf6992bb7f592c10e0f039af2f0a87d149dd1eee: Status 404 returned error can't find the container with id d8076816b63d8376b9ebda69cf6992bb7f592c10e0f039af2f0a87d149dd1eee
Jul 16 21:49:08.563538 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.563516    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7052dd71_d93b_44d0_bf9c_64005d26e75d.slice/crio-8eacea3ee440bd62204551e7ca449c09f7671a15576fef4bccaa63ebbd002179 WatchSource:0}: Error finding container 8eacea3ee440bd62204551e7ca449c09f7671a15576fef4bccaa63ebbd002179: Status 404 returned error can't find the container with id 8eacea3ee440bd62204551e7ca449c09f7671a15576fef4bccaa63ebbd002179
Jul 16 21:49:08.564898 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.564876    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda2e4c212_c11d_4dec_b3db_76d86bf09ea5.slice/crio-70e8bde4002c93344fc5b2d812acf42fc2825c4cefb4c843cf526d9869128616 WatchSource:0}: Error finding container 70e8bde4002c93344fc5b2d812acf42fc2825c4cefb4c843cf526d9869128616: Status 404 returned error can't find the container with id 70e8bde4002c93344fc5b2d812acf42fc2825c4cefb4c843cf526d9869128616
Jul 16 21:49:08.566189 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.566078    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf0d312c_dd41_4069_bd73_e9fc466f8544.slice/crio-304bf85f08ff2e9176e5bb170c2745069940741af92d5262f3157b0fd70663cd WatchSource:0}: Error finding container 304bf85f08ff2e9176e5bb170c2745069940741af92d5262f3157b0fd70663cd: Status 404 returned error can't find the container with id 304bf85f08ff2e9176e5bb170c2745069940741af92d5262f3157b0fd70663cd
Jul 16 21:49:08.567149 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:08.567127    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d7f8a0_15de_4df5_874a_b639628d4064.slice/crio-4949a7390398c4eced28acd3e752cf91dfb905dc0b246b4e49eb3aea3a2e4859 WatchSource:0}: Error finding container 4949a7390398c4eced28acd3e752cf91dfb905dc0b246b4e49eb3aea3a2e4859: Status 404 returned error can't find the container with id 4949a7390398c4eced28acd3e752cf91dfb905dc0b246b4e49eb3aea3a2e4859
Jul 16 21:49:08.628881 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.628749    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:07 +0000 UTC" deadline="2028-02-15 15:17:16.940067733 +0000 UTC"
Jul 16 21:49:08.628994 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.628887    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13889h28m8.311188653s"
Jul 16 21:49:08.670740 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.670718    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:08.670852 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:08.670834    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:08.680910 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.680886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d2cf5" event={"ID":"df0d312c-dd41-4069-bd73-e9fc466f8544","Type":"ContainerStarted","Data":"304bf85f08ff2e9176e5bb170c2745069940741af92d5262f3157b0fd70663cd"}
Jul 16 21:49:08.682487 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.682442    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wckd4" event={"ID":"a2e4c212-c11d-4dec-b3db-76d86bf09ea5","Type":"ContainerStarted","Data":"70e8bde4002c93344fc5b2d812acf42fc2825c4cefb4c843cf526d9869128616"}
Jul 16 21:49:08.684135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.684106    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h5b5" event={"ID":"55de2ac7-3161-4f49-a52c-37b655f5c8b7","Type":"ContainerStarted","Data":"d8076816b63d8376b9ebda69cf6992bb7f592c10e0f039af2f0a87d149dd1eee"}
Jul 16 21:49:08.685648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.685624    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbvhz" event={"ID":"80d51c90-35f7-4d89-8ef7-ef6667c84f03","Type":"ContainerStarted","Data":"9e274183e9a4e4083851a6fb38c6ee3c75dc3892242b50d200e3b287fbd77c9d"}
Jul 16 21:49:08.687048 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.687027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerStarted","Data":"aba30807b16e101eb9012575d3eebea9af4d0664bc72c9266945a8d7845b12ea"}
Jul 16 21:49:08.688572 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.688545    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvxqn" event={"ID":"4e959648-65d9-45ea-ac05-3522b5cc0041","Type":"ContainerStarted","Data":"5e79a8cecddd1f11e3378e21bfdadda5b1b8f7e20d9f88ae117c235c1bb4c221"}
Jul 16 21:49:08.689878 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.689859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"8eacea3ee440bd62204551e7ca449c09f7671a15576fef4bccaa63ebbd002179"}
Jul 16 21:49:08.691150 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.691133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrpqt" event={"ID":"487b236d-030e-4541-8ac0-5199b92f6b16","Type":"ContainerStarted","Data":"e450f3f31ae51a7993a2c8fcf5f157cedc0ac941a043c84d1f99f12f9d7db5fd"}
Jul 16 21:49:08.693776 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.693752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal" event={"ID":"e8d0181de62365719ca8addca6e9fe43","Type":"ContainerStarted","Data":"09e8c56ce6407b3ad6437fc52924e7f1d1a50b8221198d5244f674f09c2599d4"}
Jul 16 21:49:08.695175 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:08.695153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr" event={"ID":"91d7f8a0-15de-4df5-874a-b639628d4064","Type":"ContainerStarted","Data":"4949a7390398c4eced28acd3e752cf91dfb905dc0b246b4e49eb3aea3a2e4859"}
Jul 16 21:49:09.259652 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.259613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:09.259836 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.259666    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:09.259836 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.259813    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:09.259938 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.259872    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:11.259854009 +0000 UTC m=+6.096907807 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:09.260272 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.260254    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:09.260356 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.260305    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:11.26029077 +0000 UTC m=+6.097344550 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:09.361094 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.361056    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:09.361271 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.361218    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:09.361271 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.361241    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:09.361271 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.361253    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:09.361428 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.361311    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:11.361293024 +0000 UTC m=+6.198346805 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:09.630143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.630055    2555 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:44:07 +0000 UTC" deadline="2028-02-13 18:22:57.994557257 +0000 UTC"
Jul 16 21:49:09.630143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.630089    2555 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13844h33m48.364473346s"
Jul 16 21:49:09.674980 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.674948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:09.675139 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.675004    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:09.675139 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.675114    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:09.675787 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:09.675760    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:09.714255 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.714213    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal" event={"ID":"0dba07077d86471426887abd01210c23","Type":"ContainerDied","Data":"ee5c747f86ee499c7a7dd6cbf50fe73dfd6d106d65cda46731a1500bded7386a"}
Jul 16 21:49:09.714962 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.714180    2555 generic.go:356] "Generic (PLEG): container finished" podID="0dba07077d86471426887abd01210c23" containerID="ee5c747f86ee499c7a7dd6cbf50fe73dfd6d106d65cda46731a1500bded7386a" exitCode=0
Jul 16 21:49:09.738325 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:09.738246    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-79.ec2.internal" podStartSLOduration=2.7382269900000002 podStartE2EDuration="2.73822699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:08.721321448 +0000 UTC m=+3.558375246" watchObservedRunningTime="2026-07-16 21:49:09.73822699 +0000 UTC m=+4.575280791"
Jul 16 21:49:10.670648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:10.670608    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:10.671116 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:10.670742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:10.738938 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:10.738901    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal" event={"ID":"0dba07077d86471426887abd01210c23","Type":"ContainerStarted","Data":"3193cd7f7ca5331d9f5d75f30ed33dca7da2b7586a101633c57f336028318783"}
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:11.280954    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:11.281001    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.281169    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.281225    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:15.281208118 +0000 UTC m=+10.118261899 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.281699    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:11.281781 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.281750    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:15.281732212 +0000 UTC m=+10.118785993 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:11.381774 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:11.381742    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:11.381941 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.381913    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:11.381941 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.381935    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:11.382065 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.381949    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:11.382065 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.382004    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:15.381986532 +0000 UTC m=+10.219040329 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:11.671627 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:11.671554    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:11.672137 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.671688    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:11.672137 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:11.672043    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:11.672270 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:11.672167    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:12.670877 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:12.670848    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:12.671058 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:12.670968    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:13.672786 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:13.672719    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:13.673193 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:13.672831    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:13.673268 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:13.673216    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:13.673344 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:13.673322    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:14.670402 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:14.670363    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:14.670599 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:14.670501    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:15.315027 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:15.314989    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:15.315516 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:15.315041    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:15.315516 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.315151    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:15.315516 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.315184    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:15.315516 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.315212    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:23.315197217 +0000 UTC m=+18.152250996 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:15.315516 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.315236    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:23.315220136 +0000 UTC m=+18.152273917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:15.416650 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:15.416072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:15.416650 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.416225    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:15.416650 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.416243    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:15.416650 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.416256    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:15.416650 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.416311    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:23.416293761 +0000 UTC m=+18.253347545 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:15.672771 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:15.672700    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:15.672922 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.672832    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:15.673164 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:15.673148    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:15.673244 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:15.673220    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:16.670556 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:16.670510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:16.670952 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:16.670707    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:17.671134 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:17.670677    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:17.671134 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:17.670812    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:17.671610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:17.671237    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:17.671610 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:17.671355    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:18.670710 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:18.670673    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:18.670916 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:18.670813    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:19.670981 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:19.670894    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:19.671410 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:19.671026    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:19.671410 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:19.671083    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:19.671410 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:19.671173    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:20.670395 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:20.670361    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:20.670559 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:20.670483    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:21.670545 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:21.670506    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:21.670958 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:21.670564    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:21.670958 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:21.670638    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:21.670958 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:21.670765    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:22.670726 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:22.670694    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:22.671225 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:22.670814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:23.377359 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:23.377318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:23.377624 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:23.377372    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:23.377624 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.377508    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:23.377624 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.377566    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:23.377624 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.377596    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:39.377581713 +0000 UTC m=+34.214635504 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:23.377624 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.377612    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:49:39.377605262 +0000 UTC m=+34.214659040 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:23.478605 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:23.478574    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:23.478800 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.478780    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:23.478847 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.478806    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:23.478847 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.478819    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:23.478929 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.478879    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:49:39.478862105 +0000 UTC m=+34.315915885 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:23.671254 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:23.671173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:23.671739 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.671332    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:23.671739 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:23.671372    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:23.671739 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:23.671495    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:24.670892 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:24.670864    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:24.671043 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:24.670962    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:25.671085 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:25.671045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:25.671565 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:25.671179    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:25.671565 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:25.671240    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:25.671565 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:25.671360    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:26.669948 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:26.669920    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:26.670131 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:26.670033    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:27.670668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:27.670635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:27.671494 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:27.670635    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:27.671494 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:27.670742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:27.671494 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:27.670809    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:28.671152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.670836    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:28.671979 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:28.671262    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:28.784524 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.784424    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr" event={"ID":"91d7f8a0-15de-4df5-874a-b639628d4064","Type":"ContainerStarted","Data":"218d032c5475e26b3f550be87193e085e8482d0885947e069f495f9607554cff"}
Jul 16 21:49:28.785874 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.785850    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-d2cf5" event={"ID":"df0d312c-dd41-4069-bd73-e9fc466f8544","Type":"ContainerStarted","Data":"d1a638fc69ee57a3e610548ade2393bf380687b7d90f20e11a57dadda3803afb"}
Jul 16 21:49:28.787293 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.787271    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h5b5" event={"ID":"55de2ac7-3161-4f49-a52c-37b655f5c8b7","Type":"ContainerStarted","Data":"0606d7f193de1a11f3ed1550e5af91a0cda16419c855f4cccbe1e1317a09bba5"}
Jul 16 21:49:28.788646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.788626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbvhz" event={"ID":"80d51c90-35f7-4d89-8ef7-ef6667c84f03","Type":"ContainerStarted","Data":"8346494e20c90cd82e20d702c837a1c21c31e0c081f46a56980d81af2c3f6d34"}
Jul 16 21:49:28.790161 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.790137    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="aaa8ad5f1eb8ca04c6e2be60877d9187e49e776e6c0c7aa6d78a79861ada50c6" exitCode=0
Jul 16 21:49:28.790253 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.790204    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"aaa8ad5f1eb8ca04c6e2be60877d9187e49e776e6c0c7aa6d78a79861ada50c6"}
Jul 16 21:49:28.791697 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.791676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pvxqn" event={"ID":"4e959648-65d9-45ea-ac05-3522b5cc0041","Type":"ContainerStarted","Data":"2253452f49830a6eb269aa244f5193609c54d517cf20ac6e4a60eba07832b24a"}
Jul 16 21:49:28.794652 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"fb52c7019c3e4e5fa9b68474f2179464fc2f3f7fd59b504f96ca96d4265f9ee1"}
Jul 16 21:49:28.794763 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794741    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"08736bf5f21154150717fbf9f3084d4a2a9cdb5c3f602e76805e8aaac53a3c83"}
Jul 16 21:49:28.794862 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794805    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"8802002aa93f0810d2f46cb686143774471b5bad654cf6ac453a24179fe87f13"}
Jul 16 21:49:28.794862 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"015860ba77e2d66cba3fe81ff1c0f7847255c03e41ad0ab1f9fb81b75c3319fd"}
Jul 16 21:49:28.794951 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"f01410f7c107b416a51eb5bd6322eb60dbd9be3af259799616cbde6f29112013"}
Jul 16 21:49:28.794951 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.794931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"7588056d8b4b70f6124f37d102fabe3857b6ae39ed8b61279c9aa151b5c2bcfd"}
Jul 16 21:49:28.795897 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.795878    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wrpqt" event={"ID":"487b236d-030e-4541-8ac0-5199b92f6b16","Type":"ContainerStarted","Data":"501ff01ecda4bc95b504fbc1a3a1a5a566dbf77aa4b711987d267cb3389803d5"}
Jul 16 21:49:28.812732 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.812692    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-79.ec2.internal" podStartSLOduration=21.812681287 podStartE2EDuration="21.812681287s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:10.777061374 +0000 UTC m=+5.614115176" watchObservedRunningTime="2026-07-16 21:49:28.812681287 +0000 UTC m=+23.649735087"
Jul 16 21:49:28.850399 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.850353    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-d2cf5" podStartSLOduration=3.728898249 podStartE2EDuration="22.850339779s" totalImagesPullingTime="19.12144153s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:28.812665279 +0000 UTC m=+23.649719080" watchObservedRunningTime="2026-07-16 21:49:28.850339779 +0000 UTC m=+23.687393578"
Jul 16 21:49:28.869599 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.869555    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6h5b5" podStartSLOduration=4.691867484 podStartE2EDuration="23.869542127s" totalImagesPullingTime="19.177674643s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:28.869377931 +0000 UTC m=+23.706431732" watchObservedRunningTime="2026-07-16 21:49:28.869542127 +0000 UTC m=+23.706595928"
Jul 16 21:49:28.911265 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.911224    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pvxqn" podStartSLOduration=12.05258932 podStartE2EDuration="22.911209279s" totalImagesPullingTime="10.858619959s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:28.911051713 +0000 UTC m=+23.748105512" watchObservedRunningTime="2026-07-16 21:49:28.911209279 +0000 UTC m=+23.748263078"
Jul 16 21:49:28.911427 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.911409    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gbvhz" podStartSLOduration=4.787664789 podStartE2EDuration="23.911404285s" totalImagesPullingTime="19.123739496s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:28.890450958 +0000 UTC m=+23.727504768" watchObservedRunningTime="2026-07-16 21:49:28.911404285 +0000 UTC m=+23.748458134"
Jul 16 21:49:28.930972 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:28.930928    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wrpqt" podStartSLOduration=4.795324445 podStartE2EDuration="23.930913649s" totalImagesPullingTime="19.135589204s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:28.930431992 +0000 UTC m=+23.767485791" watchObservedRunningTime="2026-07-16 21:49:28.930913649 +0000 UTC m=+23.767967449"
Jul 16 21:49:29.118895 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.118872    2555 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:49:29.601503 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.601393    2555 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:49:29.118889803Z","UUID":"530b6a23-70a8-4119-8105-44f5b8e44701","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:49:29.602863 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.602844    2555 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:49:29.602959 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.602879    2555 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:49:29.669959 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.669937    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:29.670116 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:29.670044    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:29.670183 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.670136    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:29.670248 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:29.670207    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:29.800888 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.800853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr" event={"ID":"91d7f8a0-15de-4df5-874a-b639628d4064","Type":"ContainerStarted","Data":"587e00008b8140fc5a60e109eafd8d7e7351d9f872d563af385f71e8c5140803"}
Jul 16 21:49:29.803185 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:29.803144    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-wckd4" event={"ID":"a2e4c212-c11d-4dec-b3db-76d86bf09ea5","Type":"ContainerStarted","Data":"d9404ed717bbab698a5bfe3618bdf60731bf1fc33ba67b4cd156daeb4091919c"}
Jul 16 21:49:30.670055 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:30.670022    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:30.670271 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:30.670148    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:30.809044 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:30.808997    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr" event={"ID":"91d7f8a0-15de-4df5-874a-b639628d4064","Type":"ContainerStarted","Data":"f817131f189768b6ab7f0935be174b2eebe1d3c1080e258d547556c5d9595082"}
Jul 16 21:49:30.814626 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:30.814570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"f891c57b445d5b4f66a74bd727fcfee276ca38ad1dffb0ea1cf6b48ae726d683"}
Jul 16 21:49:30.845807 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:30.845760    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-wckd4" podStartSLOduration=5.725600291 podStartE2EDuration="24.845745837s" totalImagesPullingTime="19.120145546s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:29.823411013 +0000 UTC m=+24.660464815" watchObservedRunningTime="2026-07-16 21:49:30.845745837 +0000 UTC m=+25.682799637"
Jul 16 21:49:30.846001 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:30.845982    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-hzwhr" podStartSLOduration=3.52866865 podStartE2EDuration="24.845976934s" totalImagesPullingTime="21.317308284s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:30.845655863 +0000 UTC m=+25.682709673" watchObservedRunningTime="2026-07-16 21:49:30.845976934 +0000 UTC m=+25.683030733"
Jul 16 21:49:31.670368 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:31.670131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:31.670719 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:31.670496    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:31.670824 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:31.670132    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:31.670927 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:31.670907    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:32.229404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:32.229362    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:32.230038 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:32.230021    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:32.670832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:32.670755    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:32.670977 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:32.670859    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:32.818897 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:32.818693    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:32.819173 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:32.819153    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pvxqn"
Jul 16 21:49:33.670021 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:33.669935    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:33.670021 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:33.669988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:33.671005 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:33.670052    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:33.671005 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:33.670107    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:33.824543 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:33.824509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" event={"ID":"7052dd71-d93b-44d0-bf9c-64005d26e75d","Type":"ContainerStarted","Data":"1d58f465c97b8226bc960591e1dad320c26f1b9cf142454c1487d1ed4b6792f5"}
Jul 16 21:49:33.865277 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:33.865125    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb" podStartSLOduration=8.655432022 podStartE2EDuration="27.865111629s" totalImagesPullingTime="19.209679607s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:33.862767254 +0000 UTC m=+28.699821055" watchObservedRunningTime="2026-07-16 21:49:33.865111629 +0000 UTC m=+28.702165432"
Jul 16 21:49:34.670385 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.670354    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:34.670923 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:34.670451    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:34.829137 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.829101    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="bf1c56488c991abe53536d58377288eb107ea40b78bc9cf479f19b38f173e3aa" exitCode=0
Jul 16 21:49:34.829404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.829200    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"bf1c56488c991abe53536d58377288eb107ea40b78bc9cf479f19b38f173e3aa"}
Jul 16 21:49:34.830526 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.829987    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:34.830526 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.830071    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:34.830526 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.830113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:34.848761 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.848716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:34.850605 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:34.850583    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:49:35.673326 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.673299    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:35.673668 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:35.673396    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:35.673668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.673453    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:35.673668 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:35.673551    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:35.914303 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.914278    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwnj9"]
Jul 16 21:49:35.914422 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.914408    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:35.914566 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:35.914546    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:35.923638 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.923613    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jflr"]
Jul 16 21:49:35.923763 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.923727    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:35.923857 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:35.923838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:35.924412 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.924388    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9l6v"]
Jul 16 21:49:35.924541 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:35.924495    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:35.924628 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:35.924608    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:36.836621 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:36.836587    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="3cf5636f6eade514c3b92648f56f22651ddf7b168ceb6f954a0f7f8426804233" exitCode=0
Jul 16 21:49:36.836968 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:36.836674    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"3cf5636f6eade514c3b92648f56f22651ddf7b168ceb6f954a0f7f8426804233"}
Jul 16 21:49:37.674313 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:37.674292    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:37.674387 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:37.674335    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:37.674437 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:37.674401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:37.674437 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:37.674422    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:37.674560 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:37.674523    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:37.674643 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:37.674625    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:37.840928 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:37.840887    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="298b7807286d1142975eab59092aa2d365dd430989a56ed113444e65be6518a5" exitCode=0
Jul 16 21:49:37.841308 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:37.840946    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"298b7807286d1142975eab59092aa2d365dd430989a56ed113444e65be6518a5"}
Jul 16 21:49:39.395450 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.395358    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:39.395450 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.395420    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:39.395943 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.395508    2555 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:39.395943 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.395551    2555 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:39.395943 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.395631    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret podName:f98e5fc8-14b8-4608-a52f-8fc5efaf8778 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:11.39558783 +0000 UTC m=+66.232641612 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret") pod "global-pull-secret-syncer-jwnj9" (UID: "f98e5fc8-14b8-4608-a52f-8fc5efaf8778") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:49:39.395943 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.395657    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs podName:b5645581-2719-434f-802d-4721384c2acc nodeName:}" failed. No retries permitted until 2026-07-16 21:50:11.395646341 +0000 UTC m=+66.232700124 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs") pod "network-metrics-daemon-8jflr" (UID: "b5645581-2719-434f-802d-4721384c2acc") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:49:39.496360 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.496316    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:39.496571 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.496534    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:49:39.496571 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.496560    2555 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:49:39.496571 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.496569    2555 projected.go:196] Error preparing data for projected volume kube-api-access-knsrc for pod openshift-network-diagnostics/network-check-target-c9l6v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:39.496691 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.496626    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc podName:e46ee35a-abb6-4e2d-bf7b-d191fef4fddf nodeName:}" failed. No retries permitted until 2026-07-16 21:50:11.496611933 +0000 UTC m=+66.333665714 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-knsrc" (UniqueName: "kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc") pod "network-check-target-c9l6v" (UID: "e46ee35a-abb6-4e2d-bf7b-d191fef4fddf") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:49:39.670766 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.670697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:39.670906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.670697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:39.670906 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.670804    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-jwnj9" podUID="f98e5fc8-14b8-4608-a52f-8fc5efaf8778"
Jul 16 21:49:39.671013 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.670906    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-c9l6v" podUID="e46ee35a-abb6-4e2d-bf7b-d191fef4fddf"
Jul 16 21:49:39.671013 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:39.670697    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:39.671100 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:39.671042    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-8jflr" podUID="b5645581-2719-434f-802d-4721384c2acc"
Jul 16 21:49:40.431870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.431839    2555 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-79.ec2.internal" event="NodeReady"
Jul 16 21:49:40.432295 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.431982    2555 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:49:40.491144 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.491075    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-787fd6f8df-9jckf"]
Jul 16 21:49:40.493233 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.493202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.497786 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.497761    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:49:40.497903 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.497841    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:49:40.497964 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.497900    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:49:40.498626 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.498610    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mdgnt\""
Jul 16 21:49:40.508905 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.508718    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:49:40.510003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.509983    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zq5f8"]
Jul 16 21:49:40.511705 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.511688    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-bmwpc"]
Jul 16 21:49:40.511829 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.511813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.513421 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.513401    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.516154 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516020    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:49:40.516299 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516274    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-vlpp9\""
Jul 16 21:49:40.516523 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516502    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:49:40.516625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516564    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:49:40.516680 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516625    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5kfkb\""
Jul 16 21:49:40.516782 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:49:40.516853 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.516833    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:49:40.517669 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.517646    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-787fd6f8df-9jckf"]
Jul 16 21:49:40.529257 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.529233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq5f8"]
Jul 16 21:49:40.536854 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.536833    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmwpc"]
Jul 16 21:49:40.604264 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604232    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604399 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604280    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604399 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604348    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604546 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-cert\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.604546 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604446    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604546 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604507    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsbsz\" (UniqueName: \"kubernetes.io/projected/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-kube-api-access-tsbsz\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.604658 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604544    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604658 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604658 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604586    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.604658 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-config-volume\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.604816 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604661    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-tmp-dir\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.604816 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604709    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.604816 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604745    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g54x6\" (UniqueName: \"kubernetes.io/projected/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-kube-api-access-g54x6\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.604816 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.604772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rj9w\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.705583 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.705583 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsbsz\" (UniqueName: \"kubernetes.io/projected/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-kube-api-access-tsbsz\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705613    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705635    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705686    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-config-volume\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705710    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-tmp-dir\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705734    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705763    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g54x6\" (UniqueName: \"kubernetes.io/projected/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-kube-api-access-g54x6\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.705791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705786    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rj9w\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.706056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705829    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.706056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705852    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.706056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705884    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.706056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.705936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-cert\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.706056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.706025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.706386 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:40.706362    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:40.706453 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:40.706408    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:40.706453 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:40.706423    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:40.706453 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:40.706441    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls podName:12e3e88b-17d1-43ae-82ea-b84c3bb16fd7 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:41.206421409 +0000 UTC m=+36.043475200 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls") pod "dns-default-bmwpc" (UID: "12e3e88b-17d1-43ae-82ea-b84c3bb16fd7") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:40.706629 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:40.706500    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:41.206481809 +0000 UTC m=+36.043535602 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:40.706941 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.706914    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-tmp-dir\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.707044 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.707001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-config-volume\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.707416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.707338    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.707521 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.707489    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.710791 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.710768    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.710887 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.710808    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-cert\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.711834 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.711811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.721552 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.721529    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g54x6\" (UniqueName: \"kubernetes.io/projected/65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07-kube-api-access-g54x6\") pod \"ingress-canary-zq5f8\" (UID: \"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07\") " pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.721652 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.721534    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsbsz\" (UniqueName: \"kubernetes.io/projected/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-kube-api-access-tsbsz\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:40.722170 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.721793    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rj9w\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.722170 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.722132    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:40.824242 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.824202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zq5f8"
Jul 16 21:49:40.967038 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:40.967002    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zq5f8"]
Jul 16 21:49:40.972909 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:40.972855    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65bc0ec8_d3fa_474b_a8c7_ea4dd10c5b07.slice/crio-c20dc539da6efee6fc220cbf6e1a72e3962e16f4ed4cd17cbb788813b0efab75 WatchSource:0}: Error finding container c20dc539da6efee6fc220cbf6e1a72e3962e16f4ed4cd17cbb788813b0efab75: Status 404 returned error can't find the container with id c20dc539da6efee6fc220cbf6e1a72e3962e16f4ed4cd17cbb788813b0efab75
Jul 16 21:49:41.210573 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.210484    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:41.210738 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.210582    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:41.210738 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:41.210667    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:41.210738 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:41.210692    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:41.210738 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:41.210713    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:41.210916 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:41.210756    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:42.210737685 +0000 UTC m=+37.047791487 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:41.210916 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:41.210770    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls podName:12e3e88b-17d1-43ae-82ea-b84c3bb16fd7 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:42.210763756 +0000 UTC m=+37.047817534 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls") pod "dns-default-bmwpc" (UID: "12e3e88b-17d1-43ae-82ea-b84c3bb16fd7") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:41.670617 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.670580    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:49:41.670617 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.670595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:49:41.670617 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.670582    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:49:41.676567 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.676544    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:49:41.678649 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.678605    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:49:41.678649 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.678640    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:49:41.680925 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.680751    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:49:41.681007 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.680791    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xnvp2\""
Jul 16 21:49:41.681534 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.681518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xjnvp\""
Jul 16 21:49:41.852166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:41.852125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq5f8" event={"ID":"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07","Type":"ContainerStarted","Data":"c20dc539da6efee6fc220cbf6e1a72e3962e16f4ed4cd17cbb788813b0efab75"}
Jul 16 21:49:42.217898 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:42.217864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:42.218047 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:42.217931    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:42.218047 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:42.218032    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:42.218047 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:42.218045    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:42.218167 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:42.218050    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:42.218167 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:42.218095    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls podName:12e3e88b-17d1-43ae-82ea-b84c3bb16fd7 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:44.218081798 +0000 UTC m=+39.055135576 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls") pod "dns-default-bmwpc" (UID: "12e3e88b-17d1-43ae-82ea-b84c3bb16fd7") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:42.218167 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:42.218107    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:44.218101294 +0000 UTC m=+39.055155073 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:42.857816 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:42.857784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerStarted","Data":"d3dd4638cd715d3b86891f1eeac61ca7212abc24d06dad0518e808b004ca2e57"}
Jul 16 21:49:43.863414 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:43.863376    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="d3dd4638cd715d3b86891f1eeac61ca7212abc24d06dad0518e808b004ca2e57" exitCode=0
Jul 16 21:49:43.863887 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:43.863437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"d3dd4638cd715d3b86891f1eeac61ca7212abc24d06dad0518e808b004ca2e57"}
Jul 16 21:49:44.235481 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:44.235373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:44.235625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:44.235502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:44.235625 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:44.235575    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:44.235708 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:44.235638    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls podName:12e3e88b-17d1-43ae-82ea-b84c3bb16fd7 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:48.235621611 +0000 UTC m=+43.072675390 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls") pod "dns-default-bmwpc" (UID: "12e3e88b-17d1-43ae-82ea-b84c3bb16fd7") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:44.235708 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:44.235647    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:44.235708 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:44.235663    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:44.235794 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:44.235713    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:48.235700334 +0000 UTC m=+43.072754131 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:44.868400 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:44.868326    2555 generic.go:356] "Generic (PLEG): container finished" podID="486dac09-187b-468c-bae0-9ed832b5dd91" containerID="c594e8ea3c19b6171dc66496cdbc0d2cb40ea1617dcc8fb54fb50980cfdd89b7" exitCode=0
Jul 16 21:49:44.868400 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:44.868374    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerDied","Data":"c594e8ea3c19b6171dc66496cdbc0d2cb40ea1617dcc8fb54fb50980cfdd89b7"}
Jul 16 21:49:45.874363 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:45.874324    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" event={"ID":"486dac09-187b-468c-bae0-9ed832b5dd91","Type":"ContainerStarted","Data":"ddb5d3ff0059231589948e28af1d0401bd694d0a091a2bc7e2819a093ff5a63c"}
Jul 16 21:49:45.875773 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:45.875746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zq5f8" event={"ID":"65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07","Type":"ContainerStarted","Data":"fbc08e5e0ee0ef5c4e7ff751693fc449f8f983c29f9ccf17b4f1668aa695fa83"}
Jul 16 21:49:45.901637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:45.901588    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-fh6tz" podStartSLOduration=10.241610646 podStartE2EDuration="40.901570254s" totalImagesPullingTime="30.659959608s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:05 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:45.90051783 +0000 UTC m=+40.737571630" watchObservedRunningTime="2026-07-16 21:49:45.901570254 +0000 UTC m=+40.738624054"
Jul 16 21:49:45.922164 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:45.922091    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zq5f8" podStartSLOduration=2.006184766 podStartE2EDuration="5.922073364s" totalImagesPullingTime="3.915888598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:45.921575627 +0000 UTC m=+40.758629428" watchObservedRunningTime="2026-07-16 21:49:45.922073364 +0000 UTC m=+40.759127165"
Jul 16 21:49:47.316639 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.316603    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-czpz4"]
Jul 16 21:49:47.341833 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.341798    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.344716 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.344689    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:49:47.344875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.344727    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:49:47.344875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.344743    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:49:47.344875 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.344766    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:49:47.345015 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.344930    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czpz4"]
Jul 16 21:49:47.345232 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.345213    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vv8qm\""
Jul 16 21:49:47.461649 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.461607    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.461649 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.461651    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49dmh\" (UniqueName: \"kubernetes.io/projected/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-api-access-49dmh\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.461894 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.461695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-data-volume\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.461894 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.461732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-crio-socket\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.461894 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.461805    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563209 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563172    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563357 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563248    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563357 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:47.563305    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:49:47.563357 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:47.563356    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:48.063343061 +0000 UTC m=+42.900396839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : configmap "kube-rbac-proxy" not found
Jul 16 21:49:47.563549 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563357    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49dmh\" (UniqueName: \"kubernetes.io/projected/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-api-access-49dmh\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563549 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:47.563360    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:47.563549 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563406    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-data-volume\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563549 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:47.563502    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:48.063479988 +0000 UTC m=+42.900533780 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:47.563549 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563544    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-crio-socket\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563720 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563677    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-data-volume\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.563751 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.563733    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-crio-socket\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:47.573783 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:47.573739    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49dmh\" (UniqueName: \"kubernetes.io/projected/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-api-access-49dmh\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:48.066634 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.066595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:48.066788 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.066663    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:48.066788 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.066725    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:49:48.066788 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.066750    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:48.066877 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.066797    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:49.066782407 +0000 UTC m=+43.903836185 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : configmap "kube-rbac-proxy" not found
Jul 16 21:49:48.066877 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.066811    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:49.066805491 +0000 UTC m=+43.903859269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:48.268236 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.268203    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:48.268387 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.268271    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:48.268387 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.268371    2555 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:49:48.268387 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.268379    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:48.268496 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.268394    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:48.268496 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.268433    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls podName:12e3e88b-17d1-43ae-82ea-b84c3bb16fd7 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:56.268418732 +0000 UTC m=+51.105472512 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls") pod "dns-default-bmwpc" (UID: "12e3e88b-17d1-43ae-82ea-b84c3bb16fd7") : secret "dns-default-metrics-tls" not found
Jul 16 21:49:48.268496 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:48.268451    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:56.268441051 +0000 UTC m=+51.105494834 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:48.603312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.603281    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wdt8q"]
Jul 16 21:49:48.625375 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.625354    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wdt8q"]
Jul 16 21:49:48.625500 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.625442    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.629662 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.629632    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 16 21:49:48.629662 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.629653    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 16 21:49:48.629827 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.629722    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 16 21:49:48.629827 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.629775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 16 21:49:48.630435 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.630417    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-g6fmf\""
Jul 16 21:49:48.774182 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.774149    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-key\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.774182 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.774188    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpvtg\" (UniqueName: \"kubernetes.io/projected/7dd82af2-f79e-4496-81a0-b08147b6db55-kube-api-access-lpvtg\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.774444 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.774271    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-cabundle\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.875636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.875563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-key\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.875636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.875597    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpvtg\" (UniqueName: \"kubernetes.io/projected/7dd82af2-f79e-4496-81a0-b08147b6db55-kube-api-access-lpvtg\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.875636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.875636    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-cabundle\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.876240 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.876221    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-cabundle\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.885937 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.885913    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpvtg\" (UniqueName: \"kubernetes.io/projected/7dd82af2-f79e-4496-81a0-b08147b6db55-kube-api-access-lpvtg\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.896611 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.896590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/7dd82af2-f79e-4496-81a0-b08147b6db55-signing-key\") pod \"service-ca-58598695f7-wdt8q\" (UID: \"7dd82af2-f79e-4496-81a0-b08147b6db55\") " pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:48.934640 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:48.934615    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-58598695f7-wdt8q"
Jul 16 21:49:49.076017 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:49.075964    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-58598695f7-wdt8q"]
Jul 16 21:49:49.077340 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:49.077318    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:49.077425 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:49.077409    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:49:49.077495 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:49.077407    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:49.077556 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:49.077496    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:51.077476225 +0000 UTC m=+45.914530024 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : configmap "kube-rbac-proxy" not found
Jul 16 21:49:49.077556 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:49.077514    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:49.077661 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:49.077567    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:51.077549869 +0000 UTC m=+45.914603664 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:49.078651 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:49.078630    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dd82af2_f79e_4496_81a0_b08147b6db55.slice/crio-e46d04a99295a2917f70eafb4c6464d4784dd05fdca787308d060bcb4f6c3ab8 WatchSource:0}: Error finding container e46d04a99295a2917f70eafb4c6464d4784dd05fdca787308d060bcb4f6c3ab8: Status 404 returned error can't find the container with id e46d04a99295a2917f70eafb4c6464d4784dd05fdca787308d060bcb4f6c3ab8
Jul 16 21:49:49.887016 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:49.886976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wdt8q" event={"ID":"7dd82af2-f79e-4496-81a0-b08147b6db55","Type":"ContainerStarted","Data":"e46d04a99295a2917f70eafb4c6464d4784dd05fdca787308d060bcb4f6c3ab8"}
Jul 16 21:49:49.944153 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:49.944122    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gbvhz_80d51c90-35f7-4d89-8ef7-ef6667c84f03/dns-node-resolver/0.log"
Jul 16 21:49:50.946320 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:50.946288    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d2cf5_df0d312c-dd41-4069-bd73-e9fc466f8544/node-ca/0.log"
Jul 16 21:49:51.094008 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:51.093971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:51.094253 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:51.094057    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:51.094253 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:51.094106    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:49:51.094253 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:51.094187    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:55.094166794 +0000 UTC m=+49.931220581 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : configmap "kube-rbac-proxy" not found
Jul 16 21:49:51.094253 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:51.094200    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:51.094253 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:51.094240    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:49:55.094226547 +0000 UTC m=+49.931280328 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:51.893987 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:51.893896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-58598695f7-wdt8q" event={"ID":"7dd82af2-f79e-4496-81a0-b08147b6db55","Type":"ContainerStarted","Data":"e3d9a11b31eacba81855336d46224591930ba91fb51c2dd8b6ed0f0460828ac1"}
Jul 16 21:49:51.915304 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:51.915257    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-58598695f7-wdt8q" podStartSLOduration=1.391145882 podStartE2EDuration="3.915240481s" totalImagesPullingTime="2.524094599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:51.914520059 +0000 UTC m=+46.751573859" watchObservedRunningTime="2026-07-16 21:49:51.915240481 +0000 UTC m=+46.752294272"
Jul 16 21:49:52.147548 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:52.147446    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq5f8_65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07/serve-healthcheck-canary/0.log"
Jul 16 21:49:55.124152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:55.124106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:55.124623 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:55.124193    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:49:55.124623 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:55.124271    2555 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:49:55.124623 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:55.124297    2555 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:55.124623 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:55.124335    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.12432085 +0000 UTC m=+57.961374631 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : configmap "kube-rbac-proxy" not found
Jul 16 21:49:55.124623 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:55.124381    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls podName:7e4804b0-0f3e-45f5-87ea-6d4f28651a09 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.124362568 +0000 UTC m=+57.961416346 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls") pod "insights-runtime-extractor-czpz4" (UID: "7e4804b0-0f3e-45f5-87ea-6d4f28651a09") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:49:56.333587 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.333537    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:56.334008 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.333637    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:49:56.334008 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:56.333759    2555 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:49:56.334008 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:56.333771    2555 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-787fd6f8df-9jckf: secret "image-registry-tls" not found
Jul 16 21:49:56.334008 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:49:56.333832    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls podName:ee5ed1be-473e-4e48-8cde-5d7be7035475 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:12.333814836 +0000 UTC m=+67.170868627 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls") pod "image-registry-787fd6f8df-9jckf" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475") : secret "image-registry-tls" not found
Jul 16 21:49:56.335963 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.335942    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/12e3e88b-17d1-43ae-82ea-b84c3bb16fd7-metrics-tls\") pod \"dns-default-bmwpc\" (UID: \"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7\") " pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:56.429662 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.429631    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:56.585772 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.585702    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-bmwpc"]
Jul 16 21:49:56.588934 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:49:56.588910    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12e3e88b_17d1_43ae_82ea_b84c3bb16fd7.slice/crio-28399b3048afac4b1589db7bdd850ccb425598f2fcfd4475c1080cdb971c90df WatchSource:0}: Error finding container 28399b3048afac4b1589db7bdd850ccb425598f2fcfd4475c1080cdb971c90df: Status 404 returned error can't find the container with id 28399b3048afac4b1589db7bdd850ccb425598f2fcfd4475c1080cdb971c90df
Jul 16 21:49:56.907859 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:56.907782    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmwpc" event={"ID":"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7","Type":"ContainerStarted","Data":"28399b3048afac4b1589db7bdd850ccb425598f2fcfd4475c1080cdb971c90df"}
Jul 16 21:49:58.914092 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:58.914062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmwpc" event={"ID":"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7","Type":"ContainerStarted","Data":"42868211f8d1ea893d5b643f383162e9ce41ad95812ab80262389ef5adb88ca7"}
Jul 16 21:49:59.918031 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:59.917999    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-bmwpc" event={"ID":"12e3e88b-17d1-43ae-82ea-b84c3bb16fd7","Type":"ContainerStarted","Data":"8c74dc9ec22f0da3c5ab3c959b01a28ef57527200a299dcdf57bdbad4126d459"}
Jul 16 21:49:59.918528 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:59.918124    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:49:59.939000 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:49:59.938943    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-bmwpc" podStartSLOduration=17.752469849 podStartE2EDuration="19.938924408s" totalImagesPullingTime="2.186454559s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:49:59.937492532 +0000 UTC m=+54.774546330" watchObservedRunningTime="2026-07-16 21:49:59.938924408 +0000 UTC m=+54.775978212"
Jul 16 21:50:03.186219 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.186165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:50:03.186639 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.186272    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:50:03.186754 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.186738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:50:03.188685 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.188667    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7e4804b0-0f3e-45f5-87ea-6d4f28651a09-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-czpz4\" (UID: \"7e4804b0-0f3e-45f5-87ea-6d4f28651a09\") " pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:50:03.251710 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.251672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-czpz4"
Jul 16 21:50:03.389641 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.389608    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-czpz4"]
Jul 16 21:50:03.392874 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:03.392855    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e4804b0_0f3e_45f5_87ea_6d4f28651a09.slice/crio-375ad299878676521b2866ab11bc870a9aa0b0f37b0fd6ff4c0e8ec3672ff491 WatchSource:0}: Error finding container 375ad299878676521b2866ab11bc870a9aa0b0f37b0fd6ff4c0e8ec3672ff491: Status 404 returned error can't find the container with id 375ad299878676521b2866ab11bc870a9aa0b0f37b0fd6ff4c0e8ec3672ff491
Jul 16 21:50:03.933525 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:03.933488    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czpz4" event={"ID":"7e4804b0-0f3e-45f5-87ea-6d4f28651a09","Type":"ContainerStarted","Data":"375ad299878676521b2866ab11bc870a9aa0b0f37b0fd6ff4c0e8ec3672ff491"}
Jul 16 21:50:06.852857 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:06.852829    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-c9cwb"
Jul 16 21:50:08.949629 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:08.949586    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czpz4" event={"ID":"7e4804b0-0f3e-45f5-87ea-6d4f28651a09","Type":"ContainerStarted","Data":"e35cb3280f1b36854561489b39db7eca652428f009804607c11ebf853119a9dc"}
Jul 16 21:50:09.923144 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:09.923113    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-bmwpc"
Jul 16 21:50:11.458010 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.457971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:50:11.458010 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.458009    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:50:11.460595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.460571    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:11.461219 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.461202    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:11.471363 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.471333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f98e5fc8-14b8-4608-a52f-8fc5efaf8778-original-pull-secret\") pod \"global-pull-secret-syncer-jwnj9\" (UID: \"f98e5fc8-14b8-4608-a52f-8fc5efaf8778\") " pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:50:11.471453 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.471333    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b5645581-2719-434f-802d-4721384c2acc-metrics-certs\") pod \"network-metrics-daemon-8jflr\" (UID: \"b5645581-2719-434f-802d-4721384c2acc\") " pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:50:11.558800 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.558778    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:50:11.561861 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.561840    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:11.572363 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.572342    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:11.581589 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.581569    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knsrc\" (UniqueName: \"kubernetes.io/projected/e46ee35a-abb6-4e2d-bf7b-d191fef4fddf-kube-api-access-knsrc\") pod \"network-check-target-c9l6v\" (UID: \"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf\") " pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:50:11.686147 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.686117    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-xjnvp\""
Jul 16 21:50:11.693434 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.693407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xnvp2\""
Jul 16 21:50:11.693434 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.693430    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:50:11.694831 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.694817    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-jwnj9"
Jul 16 21:50:11.701594 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.701574    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-8jflr"
Jul 16 21:50:11.855592 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.855566    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-8jflr"]
Jul 16 21:50:11.862341 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:11.862320    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5645581_2719_434f_802d_4721384c2acc.slice/crio-894dddfe0617da9e0bf380ca5477224a5811eefc23a90d744304aaa2c901f320 WatchSource:0}: Error finding container 894dddfe0617da9e0bf380ca5477224a5811eefc23a90d744304aaa2c901f320: Status 404 returned error can't find the container with id 894dddfe0617da9e0bf380ca5477224a5811eefc23a90d744304aaa2c901f320
Jul 16 21:50:11.869860 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.869837    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-c9l6v"]
Jul 16 21:50:11.872435 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:11.872414    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode46ee35a_abb6_4e2d_bf7b_d191fef4fddf.slice/crio-b8f7aa73afb3d3b87f39032ba41c60d9eade244c0691afba5db368187302b67c WatchSource:0}: Error finding container b8f7aa73afb3d3b87f39032ba41c60d9eade244c0691afba5db368187302b67c: Status 404 returned error can't find the container with id b8f7aa73afb3d3b87f39032ba41c60d9eade244c0691afba5db368187302b67c
Jul 16 21:50:11.874709 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.874689    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-jwnj9"]
Jul 16 21:50:11.878019 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:11.877996    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf98e5fc8_14b8_4608_a52f_8fc5efaf8778.slice/crio-b79941537d931087b391708751584c8ab67d4fc7905430c4b5e942c4821aec1a WatchSource:0}: Error finding container b79941537d931087b391708751584c8ab67d4fc7905430c4b5e942c4821aec1a: Status 404 returned error can't find the container with id b79941537d931087b391708751584c8ab67d4fc7905430c4b5e942c4821aec1a
Jul 16 21:50:11.964027 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.963942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jflr" event={"ID":"b5645581-2719-434f-802d-4721384c2acc","Type":"ContainerStarted","Data":"894dddfe0617da9e0bf380ca5477224a5811eefc23a90d744304aaa2c901f320"}
Jul 16 21:50:11.965194 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.965159    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwnj9" event={"ID":"f98e5fc8-14b8-4608-a52f-8fc5efaf8778","Type":"ContainerStarted","Data":"b79941537d931087b391708751584c8ab67d4fc7905430c4b5e942c4821aec1a"}
Jul 16 21:50:11.966901 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.966870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czpz4" event={"ID":"7e4804b0-0f3e-45f5-87ea-6d4f28651a09","Type":"ContainerStarted","Data":"deef2f3a13ec1538293110b9ea374f1960296b8760c24bb91f3c108166600729"}
Jul 16 21:50:11.967941 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:11.967920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9l6v" event={"ID":"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf","Type":"ContainerStarted","Data":"b8f7aa73afb3d3b87f39032ba41c60d9eade244c0691afba5db368187302b67c"}
Jul 16 21:50:12.063704 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.063675    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"]
Jul 16 21:50:12.066061 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.066048    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:12.069067 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.069046    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 21:50:12.069259 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.069247    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-cfkrt\""
Jul 16 21:50:12.092245 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.092215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"]
Jul 16 21:50:12.129135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.129108    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787fd6f8df-9jckf"]
Jul 16 21:50:12.129328 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:12.129308    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-787fd6f8df-9jckf" podUID="ee5ed1be-473e-4e48-8cde-5d7be7035475"
Jul 16 21:50:12.163487 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.163440    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b57c7049-ec5e-49c7-84e8-b69caa8ed260-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l9jsr\" (UID: \"b57c7049-ec5e-49c7-84e8-b69caa8ed260\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:12.182849 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.182826    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:12.184831 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.184803    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.188202 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.188185    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:50:12.189397 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.189383    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:50:12.189549 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.189520    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:50:12.189668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.189651    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:50:12.189725 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.189676    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:50:12.189807 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.189781    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:50:12.190226 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.190210    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-llxxv\""
Jul 16 21:50:12.193722 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.193707    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:50:12.195977 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.195961    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-wvl26"]
Jul 16 21:50:12.197881 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.197868    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:12.203310 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.203295    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-g22nb\""
Jul 16 21:50:12.228331 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.228277    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:12.233018 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.232989    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wvl26"]
Jul 16 21:50:12.264485 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264433    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264619 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264501    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264619 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264566    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264619 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264601    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264715 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264715 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264677    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg56n\" (UniqueName: \"kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.264715 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvrlq\" (UniqueName: \"kubernetes.io/projected/b89ccd51-d8d9-439e-9219-8a1e30821fce-kube-api-access-jvrlq\") pod \"downloads-5bc67dcd46-wvl26\" (UID: \"b89ccd51-d8d9-439e-9219-8a1e30821fce\") " pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:12.264809 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.264761    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b57c7049-ec5e-49c7-84e8-b69caa8ed260-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l9jsr\" (UID: \"b57c7049-ec5e-49c7-84e8-b69caa8ed260\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:12.267245 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.267224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/b57c7049-ec5e-49c7-84e8-b69caa8ed260-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-l9jsr\" (UID: \"b57c7049-ec5e-49c7-84e8-b69caa8ed260\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:12.365385 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365352    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.365385 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365393    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365424    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365501    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365529    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg56n\" (UniqueName: \"kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365548    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvrlq\" (UniqueName: \"kubernetes.io/projected/b89ccd51-d8d9-439e-9219-8a1e30821fce-kube-api-access-jvrlq\") pod \"downloads-5bc67dcd46-wvl26\" (UID: \"b89ccd51-d8d9-439e-9219-8a1e30821fce\") " pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365606    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.365636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.365631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.366254 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.366191    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.366641 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.366596    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.366887 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.366858    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.369446 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.369423    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.369566 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.369491    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"image-registry-787fd6f8df-9jckf\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") " pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:50:12.369688 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.369668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.374447 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.374426    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:12.378448 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.378418    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvrlq\" (UniqueName: \"kubernetes.io/projected/b89ccd51-d8d9-439e-9219-8a1e30821fce-kube-api-access-jvrlq\") pod \"downloads-5bc67dcd46-wvl26\" (UID: \"b89ccd51-d8d9-439e-9219-8a1e30821fce\") " pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:12.378600 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.378583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg56n\" (UniqueName: \"kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n\") pod \"console-dcdcc6d46-6dmfc\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") " pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.496153 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.495746    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:12.505802 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.505561    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:12.556158 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.556123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"]
Jul 16 21:50:12.691222 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.691191    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:12.695084 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:12.695053    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b6700e4_a427_4477_bcca_90d855933e35.slice/crio-d040a4d4f592f6526751190924ef856e8833df2b94fe38cb53f7a013b99c70ba WatchSource:0}: Error finding container d040a4d4f592f6526751190924ef856e8833df2b94fe38cb53f7a013b99c70ba: Status 404 returned error can't find the container with id d040a4d4f592f6526751190924ef856e8833df2b94fe38cb53f7a013b99c70ba
Jul 16 21:50:12.721920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.721890    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-wvl26"]
Jul 16 21:50:12.974425 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.974390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr" event={"ID":"b57c7049-ec5e-49c7-84e8-b69caa8ed260","Type":"ContainerStarted","Data":"75b90f05f442e729d3ceb1978a7a5d284a7a5e641e3029d77896cc8fdd1de231"}
Jul 16 21:50:12.978214 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.978188    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:50:12.978503 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.978456    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcdcc6d46-6dmfc" event={"ID":"9b6700e4-a427-4477-bcca-90d855933e35","Type":"ContainerStarted","Data":"d040a4d4f592f6526751190924ef856e8833df2b94fe38cb53f7a013b99c70ba"}
Jul 16 21:50:12.986218 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:12.986195    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:50:13.059401 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:13.059373    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb89ccd51_d8d9_439e_9219_8a1e30821fce.slice/crio-fd3b52d463a14688c998b23696f6efd841a78c449e784c76cab41203511b40d1 WatchSource:0}: Error finding container fd3b52d463a14688c998b23696f6efd841a78c449e784c76cab41203511b40d1: Status 404 returned error can't find the container with id fd3b52d463a14688c998b23696f6efd841a78c449e784c76cab41203511b40d1
Jul 16 21:50:13.071858 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.071823    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.071971 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.071867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.071971 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.071896    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rj9w\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.071971 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.071950    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.072140 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.071984    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.072140 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.072022    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.072140 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.072053    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.072140 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.072106    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca\") pod \"ee5ed1be-473e-4e48-8cde-5d7be7035475\" (UID: \"ee5ed1be-473e-4e48-8cde-5d7be7035475\") "
Jul 16 21:50:13.073931 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.073662    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:13.073931 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.073886    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:50:13.074295 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.074269    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:13.075011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.074958    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:13.075119 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.075064    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:13.076166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.076123    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:13.076734 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.076703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w" (OuterVolumeSpecName: "kube-api-access-9rj9w") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "kube-api-access-9rj9w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:13.077191 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.077153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "ee5ed1be-473e-4e48-8cde-5d7be7035475" (UID: "ee5ed1be-473e-4e48-8cde-5d7be7035475"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:13.173014 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.172980    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-trusted-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173061    2555 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-installation-pull-secrets\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173077    2555 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/ee5ed1be-473e-4e48-8cde-5d7be7035475-image-registry-private-configuration\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173086    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rj9w\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-kube-api-access-9rj9w\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173096    2555 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-bound-sa-token\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173103    2555 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-certificates\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173111    2555 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/ee5ed1be-473e-4e48-8cde-5d7be7035475-registry-tls\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.173152 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.173119    2555 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/ee5ed1be-473e-4e48-8cde-5d7be7035475-ca-trust-extracted\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:13.986490 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.986385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wvl26" event={"ID":"b89ccd51-d8d9-439e-9219-8a1e30821fce","Type":"ContainerStarted","Data":"fd3b52d463a14688c998b23696f6efd841a78c449e784c76cab41203511b40d1"}
Jul 16 21:50:13.989814 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.989789    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-787fd6f8df-9jckf"
Jul 16 21:50:13.990746 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.990723    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jflr" event={"ID":"b5645581-2719-434f-802d-4721384c2acc","Type":"ContainerStarted","Data":"92af4d6c21f05b0f855e9bd8cb5f1d3e0fc55c8db23792b65018177d1a130357"}
Jul 16 21:50:13.990854 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:13.990771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-8jflr" event={"ID":"b5645581-2719-434f-802d-4721384c2acc","Type":"ContainerStarted","Data":"8b2b2b0550b82a7f77240411c0e931d17aa634fccb2c4eac09b5e24b2a0971f4"}
Jul 16 21:50:14.054380 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:14.054316    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-8jflr" podStartSLOduration=66.763176663 podStartE2EDuration="1m8.054295852s" totalImagesPullingTime="1.291119189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:14.019562293 +0000 UTC m=+68.856616118" watchObservedRunningTime="2026-07-16 21:50:14.054295852 +0000 UTC m=+68.891349655"
Jul 16 21:50:14.068281 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:14.068245    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-787fd6f8df-9jckf"]
Jul 16 21:50:14.076192 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:14.076161    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-787fd6f8df-9jckf"]
Jul 16 21:50:15.678688 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:15.678475    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5ed1be-473e-4e48-8cde-5d7be7035475" path="/var/lib/kubelet/pods/ee5ed1be-473e-4e48-8cde-5d7be7035475/volumes"
Jul 16 21:50:19.166966 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.166926    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:50:19.169637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.169297    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.178426 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.178401    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:50:19.185680 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.185624    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:50:19.223222 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223181    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223222 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223223    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223451 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223257    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223451 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223281    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223451 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223383    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223451 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223439    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8fss7\" (UniqueName: \"kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.223696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.223490    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324256 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324215    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324275    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8fss7\" (UniqueName: \"kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324300    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324343    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324371    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.324441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.324396    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.325607 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.325583    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.325795 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.325772    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.326029 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.326010    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.326213 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.326189    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.327578 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.327545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.328414 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.328389    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.334703 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.334682    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8fss7\" (UniqueName: \"kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7\") pod \"console-6c69444b76-8br98\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") " pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:19.481182 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:19.481088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:21.488299 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:21.488261    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:50:21.491863 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:21.491836    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27b8a51a_989a_42bc_9d3a_e541be960bcc.slice/crio-b4d91fc88219802b53ce72bdc5e30311322ad58f63c54c3b8c946010d14fcd7c WatchSource:0}: Error finding container b4d91fc88219802b53ce72bdc5e30311322ad58f63c54c3b8c946010d14fcd7c: Status 404 returned error can't find the container with id b4d91fc88219802b53ce72bdc5e30311322ad58f63c54c3b8c946010d14fcd7c
Jul 16 21:50:22.025668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.025572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c69444b76-8br98" event={"ID":"27b8a51a-989a-42bc-9d3a-e541be960bcc","Type":"ContainerStarted","Data":"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"}
Jul 16 21:50:22.025842 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.025671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c69444b76-8br98" event={"ID":"27b8a51a-989a-42bc-9d3a-e541be960bcc","Type":"ContainerStarted","Data":"b4d91fc88219802b53ce72bdc5e30311322ad58f63c54c3b8c946010d14fcd7c"}
Jul 16 21:50:22.027269 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.027240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr" event={"ID":"b57c7049-ec5e-49c7-84e8-b69caa8ed260","Type":"ContainerStarted","Data":"9c5d26aa54b26e1bc62faa54d2ca19af2c1539e0eef354ac999003c6519228d1"}
Jul 16 21:50:22.027434 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.027414    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:22.029118 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.029090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-jwnj9" event={"ID":"f98e5fc8-14b8-4608-a52f-8fc5efaf8778","Type":"ContainerStarted","Data":"02e4c124cf216033bfdd240dcf4ccdc2362491eb2b2816c7a1cb46ae7ba31f83"}
Jul 16 21:50:22.031012 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.030988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcdcc6d46-6dmfc" event={"ID":"9b6700e4-a427-4477-bcca-90d855933e35","Type":"ContainerStarted","Data":"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"}
Jul 16 21:50:22.032982 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.032965    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr"
Jul 16 21:50:22.033763 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.033738    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-czpz4" event={"ID":"7e4804b0-0f3e-45f5-87ea-6d4f28651a09","Type":"ContainerStarted","Data":"36f21ec9a89bb94d6f7782c7f1fbb3821d68b2e4c38e4e3ac16c013d80027e15"}
Jul 16 21:50:22.035139 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.035120    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-c9l6v" event={"ID":"e46ee35a-abb6-4e2d-bf7b-d191fef4fddf","Type":"ContainerStarted","Data":"87f49afd20b4cca882e8ebda8d65432fa2cc0a99eaa8c33c4c9052fcb0137eed"}
Jul 16 21:50:22.035259 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.035245    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:50:22.068954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.068913    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-dcdcc6d46-6dmfc" podStartSLOduration=1.450611373 podStartE2EDuration="10.068900104s" totalImagesPullingTime="8.618288731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.067782773 +0000 UTC m=+76.904836573" watchObservedRunningTime="2026-07-16 21:50:22.068900104 +0000 UTC m=+76.905953901"
Jul 16 21:50:22.069043 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.069005    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6c69444b76-8br98" podStartSLOduration=3.069000506 podStartE2EDuration="3.069000506s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.046455107 +0000 UTC m=+76.883508918" watchObservedRunningTime="2026-07-16 21:50:22.069000506 +0000 UTC m=+76.906054305"
Jul 16 21:50:22.088665 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.088625    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-czpz4" podStartSLOduration=17.284523699 podStartE2EDuration="35.088608945s" totalImagesPullingTime="17.804085246s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:47 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.087193231 +0000 UTC m=+76.924247091" watchObservedRunningTime="2026-07-16 21:50:22.088608945 +0000 UTC m=+76.925662746"
Jul 16 21:50:22.131163 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.131112    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-c9l6v" podStartSLOduration=66.678044565 podStartE2EDuration="1m16.131095669s" totalImagesPullingTime="9.453051104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.130768247 +0000 UTC m=+76.967822236" watchObservedRunningTime="2026-07-16 21:50:22.131095669 +0000 UTC m=+76.968149470"
Jul 16 21:50:22.131446 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.131421    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-jwnj9" podStartSLOduration=66.687737438 podStartE2EDuration="1m16.131413562s" totalImagesPullingTime="9.443676124s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:49:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.107351477 +0000 UTC m=+76.944405278" watchObservedRunningTime="2026-07-16 21:50:22.131413562 +0000 UTC m=+76.968467365"
Jul 16 21:50:22.149457 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.149410    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-l9jsr" podStartSLOduration=1.4112589930000001 podStartE2EDuration="10.149391498s" totalImagesPullingTime="8.738132505s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:22.148623416 +0000 UTC m=+76.985677239" watchObservedRunningTime="2026-07-16 21:50:22.149391498 +0000 UTC m=+76.986445297"
Jul 16 21:50:22.496534 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.496499    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:22.496988 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.496558    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:22.502075 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:22.502051    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:23.044800 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.044772    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:23.076133 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.076092    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"]
Jul 16 21:50:23.080651 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.080620    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.083954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.083707    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:50:23.083954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.083796    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:50:23.083954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.083857    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 21:50:23.083954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.083871    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:50:23.084243 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.084106    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 21:50:23.084508 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.084491    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2mr9k\""
Jul 16 21:50:23.104075 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.104049    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"]
Jul 16 21:50:23.152919 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.152883    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8dh4\" (UniqueName: \"kubernetes.io/projected/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-kube-api-access-j8dh4\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.153080 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.153020    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.153080 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.153043    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.153231 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.153174    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.254160 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.254126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.254333 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.254174    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.254333 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.254228    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.254333 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.254292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8dh4\" (UniqueName: \"kubernetes.io/projected/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-kube-api-access-j8dh4\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.254557 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:23.254534    2555 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 16 21:50:23.254613 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:23.254605    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls podName:ef203e7b-67b3-42d9-82be-9f4dbb20a4e6 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:23.754584943 +0000 UTC m=+78.591638743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls") pod "prometheus-operator-5945b5cfd-7kx98" (UID: "ef203e7b-67b3-42d9-82be-9f4dbb20a4e6") : secret "prometheus-operator-tls" not found
Jul 16 21:50:23.255098 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.254883    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.257716 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.257691    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.267353 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.267328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8dh4\" (UniqueName: \"kubernetes.io/projected/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-kube-api-access-j8dh4\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.757027 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.756982    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.759856 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.759813    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/ef203e7b-67b3-42d9-82be-9f4dbb20a4e6-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7kx98\" (UID: \"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:23.992637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:23.992595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"
Jul 16 21:50:24.140292 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:24.140254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7kx98"]
Jul 16 21:50:24.143499 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:24.143445    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef203e7b_67b3_42d9_82be_9f4dbb20a4e6.slice/crio-2a6278fb11c180ca342eb6bbd398312c2a3b3ed64530b99d135e4fcda41854a9 WatchSource:0}: Error finding container 2a6278fb11c180ca342eb6bbd398312c2a3b3ed64530b99d135e4fcda41854a9: Status 404 returned error can't find the container with id 2a6278fb11c180ca342eb6bbd398312c2a3b3ed64530b99d135e4fcda41854a9
Jul 16 21:50:25.048457 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:25.048416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98" event={"ID":"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6","Type":"ContainerStarted","Data":"2a6278fb11c180ca342eb6bbd398312c2a3b3ed64530b99d135e4fcda41854a9"}
Jul 16 21:50:29.482236 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:29.482122    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:29.482236 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:29.482208    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:29.487988 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:29.487964    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:30.074073 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:30.074047    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:50:30.153897 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:30.145306    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:33.082009 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:33.081961    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-wvl26" event={"ID":"b89ccd51-d8d9-439e-9219-8a1e30821fce","Type":"ContainerStarted","Data":"f499446fe3d153e4a5dc0ee268f9f14c4e69ed06211122da4d70869d1d28ff7f"}
Jul 16 21:50:33.082507 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:33.082188    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:33.097771 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:33.097743    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-wvl26"
Jul 16 21:50:33.102497 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:33.102424    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-wvl26" podStartSLOduration=1.912860372 podStartE2EDuration="21.102406222s" totalImagesPullingTime="19.18954585s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:33.101485542 +0000 UTC m=+87.938539347" watchObservedRunningTime="2026-07-16 21:50:33.102406222 +0000 UTC m=+87.939460021"
Jul 16 21:50:34.088560 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:34.088520    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98" event={"ID":"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6","Type":"ContainerStarted","Data":"b288c86bee48af5474e815059ece5fdd2954a973aaead9e817d18938218511b9"}
Jul 16 21:50:34.089123 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:34.089090    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98" event={"ID":"ef203e7b-67b3-42d9-82be-9f4dbb20a4e6","Type":"ContainerStarted","Data":"7b7b9210ba6bd521556c6b34851a66fe8d19063bd65373ac98a9821a53004632"}
Jul 16 21:50:36.479595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.479506    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7kx98" podStartSLOduration=4.180881805 podStartE2EDuration="13.479457737s" totalImagesPullingTime="9.298575932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:34.111870126 +0000 UTC m=+88.948923926" watchObservedRunningTime="2026-07-16 21:50:36.479457737 +0000 UTC m=+91.316511538"
Jul 16 21:50:36.481508 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.481452    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"]
Jul 16 21:50:36.545341 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.545305    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bw5qf"]
Jul 16 21:50:36.546851 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.546827    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.549938 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.549903    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 21:50:36.551958 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.550947    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 21:50:36.551958 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.551200    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-lrfzf\""
Jul 16 21:50:36.551958 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.551407    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:50:36.570140 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.570120    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"]
Jul 16 21:50:36.570280 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.570258    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.573009 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.572988    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:50:36.573213 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.573019    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-jjtv5\""
Jul 16 21:50:36.573323 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.573030    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:50:36.573400 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.573039    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:50:36.663454 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663419    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-sys\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.663454 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663482    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-textfile\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663576    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-metrics-client-ca\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663607    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663640    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663663    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.663712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663701    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663749    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/190d05e8-b28f-4be5-a490-3b90c3b68f1f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663803    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-wtmp\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663867    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6knz\" (UniqueName: \"kubernetes.io/projected/27c4e46d-1427-4983-86e5-f61c1601c9a8-kube-api-access-f6knz\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7chm4\" (UniqueName: \"kubernetes.io/projected/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-api-access-7chm4\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663924    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-root\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.664003 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.663951    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764408 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764321    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-wtmp\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764408 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764392    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6knz\" (UniqueName: \"kubernetes.io/projected/27c4e46d-1427-4983-86e5-f61c1601c9a8-kube-api-access-f6knz\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764446    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7chm4\" (UniqueName: \"kubernetes.io/projected/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-api-access-7chm4\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.764646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-root\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764520    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-wtmp\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.764959 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764939    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-root\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765042 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.764551    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-sys\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765042 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765002    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/27c4e46d-1427-4983-86e5-f61c1601c9a8-sys\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765042 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765015    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-textfile\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765048    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:36.765057    2555 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765072    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-metrics-client-ca\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765098    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:36.765122    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls podName:27c4e46d-1427-4983-86e5-f61c1601c9a8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:37.265102766 +0000 UTC m=+92.102156556 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls") pod "node-exporter-bw5qf" (UID: "27c4e46d-1427-4983-86e5-f61c1601c9a8") : secret "node-exporter-tls" not found
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765160    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.765201 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765190    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.765601 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765217    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.765601 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/190d05e8-b28f-4be5-a490-3b90c3b68f1f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.765601 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765354    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-textfile\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.765764 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.765655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/190d05e8-b28f-4be5-a490-3b90c3b68f1f-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.766160 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.766107    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-metrics-client-ca\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.766279 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.766183    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.766279 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.766201    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.766797 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.766778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.767711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.767522    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.768389 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.768366    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.768640 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.768617    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.791952 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.791927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6knz\" (UniqueName: \"kubernetes.io/projected/27c4e46d-1427-4983-86e5-f61c1601c9a8-kube-api-access-f6knz\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:36.812412 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.812385    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7chm4\" (UniqueName: \"kubernetes.io/projected/190d05e8-b28f-4be5-a490-3b90c3b68f1f-kube-api-access-7chm4\") pod \"kube-state-metrics-56c5594674-lm4pt\" (UID: \"190d05e8-b28f-4be5-a490-3b90c3b68f1f\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:36.859837 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:36.859783    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"
Jul 16 21:50:37.116256 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.115486    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-lm4pt"]
Jul 16 21:50:37.121817 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:37.121788    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190d05e8_b28f_4be5_a490_3b90c3b68f1f.slice/crio-1f588f806b6bebb26f8305a9cacb86095af7e03c69a5a13e53cbfc1058c4f088 WatchSource:0}: Error finding container 1f588f806b6bebb26f8305a9cacb86095af7e03c69a5a13e53cbfc1058c4f088: Status 404 returned error can't find the container with id 1f588f806b6bebb26f8305a9cacb86095af7e03c69a5a13e53cbfc1058c4f088
Jul 16 21:50:37.269269 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.269235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:37.272175 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.272145    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/27c4e46d-1427-4983-86e5-f61c1601c9a8-node-exporter-tls\") pod \"node-exporter-bw5qf\" (UID: \"27c4e46d-1427-4983-86e5-f61c1601c9a8\") " pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:37.483209 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.483131    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bw5qf"
Jul 16 21:50:37.532829 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:37.532790    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27c4e46d_1427_4983_86e5_f61c1601c9a8.slice/crio-045d70bb7969f24dcabc88e99d1bd3cb8e98fc0c86b3d0664f4d30fd0731dba9 WatchSource:0}: Error finding container 045d70bb7969f24dcabc88e99d1bd3cb8e98fc0c86b3d0664f4d30fd0731dba9: Status 404 returned error can't find the container with id 045d70bb7969f24dcabc88e99d1bd3cb8e98fc0c86b3d0664f4d30fd0731dba9
Jul 16 21:50:37.621832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.621800    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:50:37.650708 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.650678    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.657034 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.657011    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 21:50:37.658052 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.657871    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 21:50:37.658052 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.657954    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 21:50:37.659261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.658260    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 21:50:37.659261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.658351    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-htwfh\""
Jul 16 21:50:37.659261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.658627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 21:50:37.659261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.658892    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 21:50:37.659261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.659096    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 21:50:37.665526 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.665507    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 21:50:37.672583 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.672564    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 21:50:37.680398 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.680374    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:50:37.777015 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.776981    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777044    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777073    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z98w8\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-kube-api-access-z98w8\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777102    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777144    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777208    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777267    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777305    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777381 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777366    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777700 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777390    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.777700 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.777430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878275 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878235    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878285    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878351    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878380    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878404    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878480 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878440    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878519    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878535    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z98w8\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-kube-api-access-z98w8\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878553    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878580    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.878623    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.878832 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:37.878806    2555 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 16 21:50:37.879148 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:37.878876    2555 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls podName:6dd0018c-c738-473e-bca1-670a807161b1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.378855032 +0000 UTC m=+93.215908834 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "6dd0018c-c738-473e-bca1-670a807161b1") : secret "alertmanager-main-tls" not found
Jul 16 21:50:37.880491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.879724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.880491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.880359    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.881456 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.881357    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6dd0018c-c738-473e-bca1-670a807161b1-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.884631 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.883045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6dd0018c-c738-473e-bca1-670a807161b1-config-out\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.884631 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.883606    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-tls-assets\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.884631 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.884590    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.884987 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.884969    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.885212 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.885190    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-web-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.885694 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.885447    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.887191 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.887166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-config-volume\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.895271 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.895224    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z98w8\" (UniqueName: \"kubernetes.io/projected/6dd0018c-c738-473e-bca1-670a807161b1-kube-api-access-z98w8\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:37.904966 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:37.904937    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:38.109952 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:38.109865    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt" event={"ID":"190d05e8-b28f-4be5-a490-3b90c3b68f1f","Type":"ContainerStarted","Data":"1f588f806b6bebb26f8305a9cacb86095af7e03c69a5a13e53cbfc1058c4f088"}
Jul 16 21:50:38.111417 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:38.111387    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw5qf" event={"ID":"27c4e46d-1427-4983-86e5-f61c1601c9a8","Type":"ContainerStarted","Data":"045d70bb7969f24dcabc88e99d1bd3cb8e98fc0c86b3d0664f4d30fd0731dba9"}
Jul 16 21:50:38.383339 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:38.383166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:38.386354 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:38.386322    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/6dd0018c-c738-473e-bca1-670a807161b1-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"6dd0018c-c738-473e-bca1-670a807161b1\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:38.563455 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:38.563418    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:50:39.871079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:39.870801    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:50:39.883235 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:39.878508    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd0018c_c738_473e_bca1_670a807161b1.slice/crio-e10b45355d5e5259946c87f01c444d0aee6d4f68d93091c88a6f417503365408 WatchSource:0}: Error finding container e10b45355d5e5259946c87f01c444d0aee6d4f68d93091c88a6f417503365408: Status 404 returned error can't find the container with id e10b45355d5e5259946c87f01c444d0aee6d4f68d93091c88a6f417503365408
Jul 16 21:50:40.124286 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.124247    2555 generic.go:356] "Generic (PLEG): container finished" podID="27c4e46d-1427-4983-86e5-f61c1601c9a8" containerID="6ef76d942a6f9492a83842fdedf260260d6e7eb3b28a770f800c9d5946fc0489" exitCode=0
Jul 16 21:50:40.124628 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.124600    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw5qf" event={"ID":"27c4e46d-1427-4983-86e5-f61c1601c9a8","Type":"ContainerDied","Data":"6ef76d942a6f9492a83842fdedf260260d6e7eb3b28a770f800c9d5946fc0489"}
Jul 16 21:50:40.129896 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.129611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"e10b45355d5e5259946c87f01c444d0aee6d4f68d93091c88a6f417503365408"}
Jul 16 21:50:40.132870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.132172    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt" event={"ID":"190d05e8-b28f-4be5-a490-3b90c3b68f1f","Type":"ContainerStarted","Data":"a7daf8eca809562247ef3facad002ee1562a80420a153adfb31340b1f33653d3"}
Jul 16 21:50:40.132870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.132230    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt" event={"ID":"190d05e8-b28f-4be5-a490-3b90c3b68f1f","Type":"ContainerStarted","Data":"be9f40912df54022bab8ecb5adc544c37e8ca8675d9013c2057cb24b617eac9e"}
Jul 16 21:50:40.132870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:40.132257    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt" event={"ID":"190d05e8-b28f-4be5-a490-3b90c3b68f1f","Type":"ContainerStarted","Data":"a6164618187e72c8a7552d28de2e574234ed434383e2f52efd69c55f673dd735"}
Jul 16 21:50:41.148091 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.148053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw5qf" event={"ID":"27c4e46d-1427-4983-86e5-f61c1601c9a8","Type":"ContainerStarted","Data":"28713153f0e637aae4c3639f6f7f3790665c04e9016140104170e2f83c52cc41"}
Jul 16 21:50:41.148663 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.148117    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bw5qf" event={"ID":"27c4e46d-1427-4983-86e5-f61c1601c9a8","Type":"ContainerStarted","Data":"5653f57f6ab5297031a8666e44782717546443c4a3f077adb6bcc82fedf89643"}
Jul 16 21:50:41.186563 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.186501    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-lm4pt" podStartSLOduration=2.631512395 podStartE2EDuration="5.186450134s" totalImagesPullingTime="2.554937739s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:40.188989469 +0000 UTC m=+95.026043270" watchObservedRunningTime="2026-07-16 21:50:41.186450134 +0000 UTC m=+96.023503935"
Jul 16 21:50:41.186778 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.186637    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bw5qf" podStartSLOduration=2.898568763 podStartE2EDuration="5.186626934s" totalImagesPullingTime="2.144717971s" totalInitContainerRuntime="143.3402ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:41.184949434 +0000 UTC m=+96.022003270" watchObservedRunningTime="2026-07-16 21:50:41.186626934 +0000 UTC m=+96.023680735"
Jul 16 21:50:41.212301 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.212272    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"]
Jul 16 21:50:41.235241 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.234661    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"]
Jul 16 21:50:41.235241 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.234787    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.237922 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.237895    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 21:50:41.238271 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.238246    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-fgwsv\""
Jul 16 21:50:41.412274 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.412187    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97b12f50-174c-42ec-b704-f54055e78d09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.412274 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.412239    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r2gp\" (UniqueName: \"kubernetes.io/projected/97b12f50-174c-42ec-b704-f54055e78d09-kube-api-access-5r2gp\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.513165 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.513138    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97b12f50-174c-42ec-b704-f54055e78d09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.513280 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.513177    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r2gp\" (UniqueName: \"kubernetes.io/projected/97b12f50-174c-42ec-b704-f54055e78d09-kube-api-access-5r2gp\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.515854 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.515827    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/97b12f50-174c-42ec-b704-f54055e78d09-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.522353 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.522327    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r2gp\" (UniqueName: \"kubernetes.io/projected/97b12f50-174c-42ec-b704-f54055e78d09-kube-api-access-5r2gp\") pod \"monitoring-plugin-7b7b7bc8bf-q7bb2\" (UID: \"97b12f50-174c-42ec-b704-f54055e78d09\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.547801 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.547782    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:41.664116 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.664025    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"]
Jul 16 21:50:41.683915 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.682840    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.685748    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.685966    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.686270    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.686530    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.686710    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 21:50:41.688496 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.686943    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-7dvg7\""
Jul 16 21:50:41.698389 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.698358    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 21:50:41.698932 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.698629    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"]
Jul 16 21:50:41.747608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.747566    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"]
Jul 16 21:50:41.752066 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:41.752038    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b12f50_174c_42ec_b704_f54055e78d09.slice/crio-966150e1797305d89f12bbc433d2c8a3ebeb68bed89cf46ffd65b509a96215ee WatchSource:0}: Error finding container 966150e1797305d89f12bbc433d2c8a3ebeb68bed89cf46ffd65b509a96215ee: Status 404 returned error can't find the container with id 966150e1797305d89f12bbc433d2c8a3ebeb68bed89cf46ffd65b509a96215ee
Jul 16 21:50:41.816562 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816527    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcj2f\" (UniqueName: \"kubernetes.io/projected/22b26901-2951-410a-9867-058784aae2d9-kube-api-access-rcj2f\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816562 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816567    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-metrics-client-ca\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816608    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816690    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-serving-certs-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816753    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-telemeter-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816794    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.816825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816821    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-federate-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.817015 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.816876    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918372 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918299    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-metrics-client-ca\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918372 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918348    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918381    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-serving-certs-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918416    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-telemeter-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918448    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918498    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-federate-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918618    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.918863 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.918658    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcj2f\" (UniqueName: \"kubernetes.io/projected/22b26901-2951-410a-9867-058784aae2d9-kube-api-access-rcj2f\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.919216 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.919185    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-serving-certs-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.919415 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.919380    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-telemeter-trusted-ca-bundle\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.919898 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.919874    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/22b26901-2951-410a-9867-058784aae2d9-metrics-client-ca\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.921105 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.921080    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.921351 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.921328    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-secret-telemeter-client\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.921672 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.921657    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-federate-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.921719 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.921711    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/22b26901-2951-410a-9867-058784aae2d9-telemeter-client-tls\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:41.929194 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:41.929163    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcj2f\" (UniqueName: \"kubernetes.io/projected/22b26901-2951-410a-9867-058784aae2d9-kube-api-access-rcj2f\") pod \"telemeter-client-767f5f56bc-2d8qn\" (UID: \"22b26901-2951-410a-9867-058784aae2d9\") " pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:42.007905 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:42.007874    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"
Jul 16 21:50:42.153286 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:42.153248    2555 generic.go:356] "Generic (PLEG): container finished" podID="6dd0018c-c738-473e-bca1-670a807161b1" containerID="e695e995e4e38aaa333102ad002794d39845c29923174fc86c4cb12d7faa9ba0" exitCode=0
Jul 16 21:50:42.153755 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:42.153327    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerDied","Data":"e695e995e4e38aaa333102ad002794d39845c29923174fc86c4cb12d7faa9ba0"}
Jul 16 21:50:42.154669 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:42.154650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2" event={"ID":"97b12f50-174c-42ec-b704-f54055e78d09","Type":"ContainerStarted","Data":"966150e1797305d89f12bbc433d2c8a3ebeb68bed89cf46ffd65b509a96215ee"}
Jul 16 21:50:42.163441 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:42.163425    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-767f5f56bc-2d8qn"]
Jul 16 21:50:42.165317 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:42.165296    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b26901_2951_410a_9867_058784aae2d9.slice/crio-b71bcf74664c84a8f443bd2e12a4c2cdea4a6fe90e9c6f3610b655d1065ad9e9 WatchSource:0}: Error finding container b71bcf74664c84a8f443bd2e12a4c2cdea4a6fe90e9c6f3610b655d1065ad9e9: Status 404 returned error can't find the container with id b71bcf74664c84a8f443bd2e12a4c2cdea4a6fe90e9c6f3610b655d1065ad9e9
Jul 16 21:50:43.164919 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:43.164830    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn" event={"ID":"22b26901-2951-410a-9867-058784aae2d9","Type":"ContainerStarted","Data":"b71bcf74664c84a8f443bd2e12a4c2cdea4a6fe90e9c6f3610b655d1065ad9e9"}
Jul 16 21:50:46.181696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.181660    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn" event={"ID":"22b26901-2951-410a-9867-058784aae2d9","Type":"ContainerStarted","Data":"2b570c5b0a6948d186061fa3fb5784bdeb45a67b2e78ca71296db177c81eb366"}
Jul 16 21:50:46.182195 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.181757    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn" event={"ID":"22b26901-2951-410a-9867-058784aae2d9","Type":"ContainerStarted","Data":"c42e7d9ea5f29560bb5b1b81b92dcdd69ac8243d28944b226811f89df4248c91"}
Jul 16 21:50:46.182195 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.181789    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn" event={"ID":"22b26901-2951-410a-9867-058784aae2d9","Type":"ContainerStarted","Data":"ed789fe7bb1d46933e7ec858469476910e8a36e96f574ff2fe4cedd61a5f4e87"}
Jul 16 21:50:46.184559 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.184538    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"28f2b1b4f2f87dc6a07705472f119d6aa5ac513199985b03d1e2f8c742338db3"}
Jul 16 21:50:46.184674 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.184601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"e85d3a5749e19550a724079f1372c2af351380e0a4f60f0284409ee2e83d8871"}
Jul 16 21:50:46.184674 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.184642    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"99c5853d1ceb8d7176e24bb1b7cae22db278207695b5078db6b7d8e6b1c5c754"}
Jul 16 21:50:46.184757 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.184676    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"0a682d3cfc17e97e7c766639049bff84b59d39e61963edf1f7cb7a9a4d521c2b"}
Jul 16 21:50:46.184757 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.184710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"6868ca54b199dc0414c0163a1845964ba09ba97b2df968eb42ce2c94ee01fd53"}
Jul 16 21:50:46.185873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.185853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2" event={"ID":"97b12f50-174c-42ec-b704-f54055e78d09","Type":"ContainerStarted","Data":"7b231cacfa0163a1ff58fb275ce97444a6a2e456bc82b32b7cab473a31b30c8f"}
Jul 16 21:50:46.186067 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.186051    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:46.190968 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.190951    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2"
Jul 16 21:50:46.209635 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.209069    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-767f5f56bc-2d8qn" podStartSLOduration=2.225838508 podStartE2EDuration="5.209052471s" totalImagesPullingTime="2.983213963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:46.208222684 +0000 UTC m=+101.045276485" watchObservedRunningTime="2026-07-16 21:50:46.209052471 +0000 UTC m=+101.046106262"
Jul 16 21:50:46.226243 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:46.226132    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-q7bb2" podStartSLOduration=1.833991253 podStartE2EDuration="5.2261147s" totalImagesPullingTime="3.392123447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:46.224899862 +0000 UTC m=+101.061953661" watchObservedRunningTime="2026-07-16 21:50:46.2261147 +0000 UTC m=+101.063168501"
Jul 16 21:50:47.077040 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.077015    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:50:47.100810 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.100782    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:50:47.100928 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.100906    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.195972 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.195932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"6dd0018c-c738-473e-bca1-670a807161b1","Type":"ContainerStarted","Data":"f6c088cc04e723b074a8394cd74d512c8ce7861cfdfffd20c76a3add114424ec"}
Jul 16 21:50:47.228714 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.228661    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.366934618 podStartE2EDuration="10.228643537s" totalImagesPullingTime="5.843019455s" totalInitContainerRuntime="18.689464ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:37 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:47.227194951 +0000 UTC m=+102.064248754" watchObservedRunningTime="2026-07-16 21:50:47.228643537 +0000 UTC m=+102.065697338"
Jul 16 21:50:47.268446 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.268410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269066 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269038    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2hrt\" (UniqueName: \"kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269300 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269276    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269427 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269411    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269563 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269547    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269716 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.269825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.269804    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.370784    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.370951    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.370986    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2hrt\" (UniqueName: \"kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.371107    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.371143    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.371167    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.371491 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.371233    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.372014 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.371983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.372129 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.372025    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.372129 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.372122    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.372420 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.372155    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.374071 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.374050    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.374071 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.374069    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.384983 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.384959    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2hrt\" (UniqueName: \"kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt\") pod \"console-7569495f5c-hs8xn\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") " pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.412701 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.412672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:47.625647 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:47.625621    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:50:47.626449 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:50:47.626427    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c939d39_59f9_4f9b_b05c_f9d5a781a7e1.slice/crio-a4e55fcd3a11268864e7c5c89dcd6b57d91b2c1393c44e97bbc82c1d55483954 WatchSource:0}: Error finding container a4e55fcd3a11268864e7c5c89dcd6b57d91b2c1393c44e97bbc82c1d55483954: Status 404 returned error can't find the container with id a4e55fcd3a11268864e7c5c89dcd6b57d91b2c1393c44e97bbc82c1d55483954
Jul 16 21:50:48.201098 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:48.201064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7569495f5c-hs8xn" event={"ID":"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1","Type":"ContainerStarted","Data":"148c7a3601df0c8b196026aa5923c1b44dd2de6c187389f9fcaceb2a8de99952"}
Jul 16 21:50:48.201578 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:48.201114    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7569495f5c-hs8xn" event={"ID":"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1","Type":"ContainerStarted","Data":"a4e55fcd3a11268864e7c5c89dcd6b57d91b2c1393c44e97bbc82c1d55483954"}
Jul 16 21:50:48.224497 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:48.224439    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7569495f5c-hs8xn" podStartSLOduration=1.224424118 podStartE2EDuration="1.224424118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:48.222605481 +0000 UTC m=+103.059659280" watchObservedRunningTime="2026-07-16 21:50:48.224424118 +0000 UTC m=+103.061477918"
Jul 16 21:50:53.042257 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:53.042228    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-c9l6v"
Jul 16 21:50:55.184902 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.184849    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-dcdcc6d46-6dmfc" podUID="9b6700e4-a427-4477-bcca-90d855933e35" containerName="console" containerID="cri-o://bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf" gracePeriod=15
Jul 16 21:50:55.450672 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.450607    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:55.532310 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532282    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532501 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532330    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532501 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532353    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532630 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532502    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532689 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532630    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zg56n\" (UniqueName: \"kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532689 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532670    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert\") pod \"9b6700e4-a427-4477-bcca-90d855933e35\" (UID: \"9b6700e4-a427-4477-bcca-90d855933e35\") "
Jul 16 21:50:55.532788 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532730    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca" (OuterVolumeSpecName: "service-ca") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:55.532788 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532739    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config" (OuterVolumeSpecName: "console-config") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:55.532971 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532902    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-console-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:55.532971 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.532970    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:55.533108 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.533055    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:50:55.534810 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.534788    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:55.534950 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.534930    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n" (OuterVolumeSpecName: "kube-api-access-zg56n") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "kube-api-access-zg56n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:50:55.534950 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.534941    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9b6700e4-a427-4477-bcca-90d855933e35" (UID: "9b6700e4-a427-4477-bcca-90d855933e35"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:50:55.634308 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.634281    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-oauth-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:55.634308 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.634302    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9b6700e4-a427-4477-bcca-90d855933e35-console-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:55.634308 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.634312    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zg56n\" (UniqueName: \"kubernetes.io/projected/9b6700e4-a427-4477-bcca-90d855933e35-kube-api-access-zg56n\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:55.634556 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:55.634321    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9b6700e4-a427-4477-bcca-90d855933e35-oauth-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:50:56.234709 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.234674    2555 generic.go:356] "Generic (PLEG): container finished" podID="9b6700e4-a427-4477-bcca-90d855933e35" containerID="bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf" exitCode=0
Jul 16 21:50:56.234709 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.234715    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-dcdcc6d46-6dmfc"
Jul 16 21:50:56.235233 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.234737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcdcc6d46-6dmfc" event={"ID":"9b6700e4-a427-4477-bcca-90d855933e35","Type":"ContainerDied","Data":"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"}
Jul 16 21:50:56.235233 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.234785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-dcdcc6d46-6dmfc" event={"ID":"9b6700e4-a427-4477-bcca-90d855933e35","Type":"ContainerDied","Data":"d040a4d4f592f6526751190924ef856e8833df2b94fe38cb53f7a013b99c70ba"}
Jul 16 21:50:56.235233 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.234814    2555 scope.go:117] "RemoveContainer" containerID="bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"
Jul 16 21:50:56.245111 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.245090    2555 scope.go:117] "RemoveContainer" containerID="bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"
Jul 16 21:50:56.245619 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:50:56.245592    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf\": container with ID starting with bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf not found: ID does not exist" containerID="bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"
Jul 16 21:50:56.245702 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.245630    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf"} err="failed to get container status \"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf\": rpc error: code = NotFound desc = could not find container \"bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf\": container with ID starting with bb880aa16c2e4e668abd1ca8dcf6134bd16e8113b232519ad9023aa85a7fc6cf not found: ID does not exist"
Jul 16 21:50:56.253784 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.253764    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:56.257981 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:56.257962    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-dcdcc6d46-6dmfc"]
Jul 16 21:50:57.413319 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:57.413290    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:57.413815 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:57.413352    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:57.417892 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:57.417867    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:57.676004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:57.675922    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b6700e4-a427-4477-bcca-90d855933e35" path="/var/lib/kubelet/pods/9b6700e4-a427-4477-bcca-90d855933e35/volumes"
Jul 16 21:50:58.246387 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:58.246353    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:50:58.298457 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:50:58.298426    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:51:07.458581 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:07.458545    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq5f8_65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07/serve-healthcheck-canary/0.log"
Jul 16 21:51:23.319164 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.319117    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6c69444b76-8br98" podUID="27b8a51a-989a-42bc-9d3a-e541be960bcc" containerName="console" containerID="cri-o://0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330" gracePeriod=15
Jul 16 21:51:23.609607 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.609579    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:51:23.672920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.672888    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673087 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.672935    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8fss7\" (UniqueName: \"kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673087 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.672969    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673087 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673006    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673087 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673042    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673276 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673095    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.673276 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle\") pod \"27b8a51a-989a-42bc-9d3a-e541be960bcc\" (UID: \"27b8a51a-989a-42bc-9d3a-e541be960bcc\") "
Jul 16 21:51:23.674348 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.674348 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.673867    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.674562 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.674416    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config" (OuterVolumeSpecName: "console-config") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.678004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.675332    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca" (OuterVolumeSpecName: "service-ca") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:23.678004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.675511    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-oauth-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.678004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.675609    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.678004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.675624    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-trusted-ca-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.678004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.676418    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:23.678307 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.678175    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7" (OuterVolumeSpecName: "kube-api-access-8fss7") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "kube-api-access-8fss7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:23.678307 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.678202    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "27b8a51a-989a-42bc-9d3a-e541be960bcc" (UID: "27b8a51a-989a-42bc-9d3a-e541be960bcc"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:23.776915 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.776879    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/27b8a51a-989a-42bc-9d3a-e541be960bcc-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.776915 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.776911    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8fss7\" (UniqueName: \"kubernetes.io/projected/27b8a51a-989a-42bc-9d3a-e541be960bcc-kube-api-access-8fss7\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.777113 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.776923    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:23.777113 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:23.776948    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/27b8a51a-989a-42bc-9d3a-e541be960bcc-console-oauth-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:51:24.354612 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.354576    2555 generic.go:356] "Generic (PLEG): container finished" podID="27b8a51a-989a-42bc-9d3a-e541be960bcc" containerID="0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330" exitCode=0
Jul 16 21:51:24.355012 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.354645    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6c69444b76-8br98"
Jul 16 21:51:24.355012 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.354661    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c69444b76-8br98" event={"ID":"27b8a51a-989a-42bc-9d3a-e541be960bcc","Type":"ContainerDied","Data":"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"}
Jul 16 21:51:24.355012 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.354717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6c69444b76-8br98" event={"ID":"27b8a51a-989a-42bc-9d3a-e541be960bcc","Type":"ContainerDied","Data":"b4d91fc88219802b53ce72bdc5e30311322ad58f63c54c3b8c946010d14fcd7c"}
Jul 16 21:51:24.355012 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.354738    2555 scope.go:117] "RemoveContainer" containerID="0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"
Jul 16 21:51:24.363026 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.363006    2555 scope.go:117] "RemoveContainer" containerID="0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"
Jul 16 21:51:24.363327 ip-10-0-131-79 kubenswrapper[2555]: E0716 21:51:24.363303    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330\": container with ID starting with 0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330 not found: ID does not exist" containerID="0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"
Jul 16 21:51:24.363390 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.363339    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330"} err="failed to get container status \"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330\": rpc error: code = NotFound desc = could not find container \"0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330\": container with ID starting with 0cadba3f8538b1487d5128475403210af09b418c1f130ae3b800d1f8bf12d330 not found: ID does not exist"
Jul 16 21:51:24.382602 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.382575    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:51:24.387525 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:24.387501    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6c69444b76-8br98"]
Jul 16 21:51:25.675351 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:51:25.675319    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="27b8a51a-989a-42bc-9d3a-e541be960bcc" path="/var/lib/kubelet/pods/27b8a51a-989a-42bc-9d3a-e541be960bcc/volumes"
Jul 16 21:52:02.099226 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099151    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099495    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="27b8a51a-989a-42bc-9d3a-e541be960bcc" containerName="console"
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099510    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="27b8a51a-989a-42bc-9d3a-e541be960bcc" containerName="console"
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099519    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b6700e4-a427-4477-bcca-90d855933e35" containerName="console"
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099524    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b6700e4-a427-4477-bcca-90d855933e35" containerName="console"
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099574    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="27b8a51a-989a-42bc-9d3a-e541be960bcc" containerName="console"
Jul 16 21:52:02.099667 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.099581    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b6700e4-a427-4477-bcca-90d855933e35" containerName="console"
Jul 16 21:52:02.101559 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.101542    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.116232 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.116208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:52:02.172981 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.172953    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.172981 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.172987    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.173211 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.173015    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.173211 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.173098    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whhbm\" (UniqueName: \"kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.173211 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.173147    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.173211 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.173163    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.173211 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.173199    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273565    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273660    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whhbm\" (UniqueName: \"kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273693    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273708    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273739    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.273870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.273768    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.274584 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.274535    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.274584 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.274558    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.274751 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.274637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.274751 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.274668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.276271 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.276248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.276367 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.276306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.282386 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.282367    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whhbm\" (UniqueName: \"kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm\") pod \"console-58d7d65fc4-zpkwt\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") " pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.410209 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.410124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:02.544929 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:02.544740    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:52:02.547615 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:52:02.547588    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53bfcfee_81ee_460c_9b83_3602082799c0.slice/crio-f1302b60251d283b41f16663d2a4d7584ce934c73cd5fca84ad261ca3aace728 WatchSource:0}: Error finding container f1302b60251d283b41f16663d2a4d7584ce934c73cd5fca84ad261ca3aace728: Status 404 returned error can't find the container with id f1302b60251d283b41f16663d2a4d7584ce934c73cd5fca84ad261ca3aace728
Jul 16 21:52:03.501365 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:03.501326    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d7d65fc4-zpkwt" event={"ID":"53bfcfee-81ee-460c-9b83-3602082799c0","Type":"ContainerStarted","Data":"b089e0b113228740cd82a6b27f10c7d348bee60b89494caf44809f61df695f23"}
Jul 16 21:52:03.501768 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:03.501385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d7d65fc4-zpkwt" event={"ID":"53bfcfee-81ee-460c-9b83-3602082799c0","Type":"ContainerStarted","Data":"f1302b60251d283b41f16663d2a4d7584ce934c73cd5fca84ad261ca3aace728"}
Jul 16 21:52:03.520413 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:03.520368    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-58d7d65fc4-zpkwt" podStartSLOduration=1.520351464 podStartE2EDuration="1.520351464s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:03.518919366 +0000 UTC m=+178.355973167" watchObservedRunningTime="2026-07-16 21:52:03.520351464 +0000 UTC m=+178.357405263"
Jul 16 21:52:12.410636 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:12.410589    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:12.411160 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:12.410655    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:12.415448 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:12.415426    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:12.538617 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:12.538593    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:52:12.592727 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:12.592699    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:52:33.065194 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.065160    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"]
Jul 16 21:52:33.067622 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.067601    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.069985 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.069961    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:52:33.070666 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.070651    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggr5m\""
Jul 16 21:52:33.070721 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.070651    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:52:33.078494 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.078454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"]
Jul 16 21:52:33.241062 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.241025    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.241062 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.241066    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.241293 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.241106    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7z8fz\" (UniqueName: \"kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.342217 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.342136    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.342217 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.342171    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.342404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.342385    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7z8fz\" (UniqueName: \"kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.342502 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.342486    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.342608 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.342582    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.350930 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.350905    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7z8fz\" (UniqueName: \"kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz\") pod \"685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") " pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.376853 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.376830    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:33.505415 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.505381    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"]
Jul 16 21:52:33.508733 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:52:33.508707    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod841a5ddb_bf51_4179_aafe_ebd6e76373c8.slice/crio-3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032 WatchSource:0}: Error finding container 3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032: Status 404 returned error can't find the container with id 3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032
Jul 16 21:52:33.616644 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:33.616561    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd" event={"ID":"841a5ddb-bf51-4179-aafe-ebd6e76373c8","Type":"ContainerStarted","Data":"3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032"}
Jul 16 21:52:37.613071 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:37.613007    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-7569495f5c-hs8xn" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerName="console" containerID="cri-o://148c7a3601df0c8b196026aa5923c1b44dd2de6c187389f9fcaceb2a8de99952" gracePeriod=15
Jul 16 21:52:38.243673 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.243626    2555 patch_prober.go:28] interesting pod/console-7569495f5c-hs8xn container/console namespace/openshift-console: Readiness probe status=failure output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused" start-of-body=
Jul 16 21:52:38.243861 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.243704    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/console-7569495f5c-hs8xn" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerName="console" probeResult="failure" output="Get \"https://10.134.0.20:8443/health\": dial tcp 10.134.0.20:8443: connect: connection refused"
Jul 16 21:52:38.640458 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.640434    2555 generic.go:356] "Generic (PLEG): container finished" podID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerID="148c7a3601df0c8b196026aa5923c1b44dd2de6c187389f9fcaceb2a8de99952" exitCode=0
Jul 16 21:52:38.640894 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.640496    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7569495f5c-hs8xn" event={"ID":"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1","Type":"ContainerDied","Data":"148c7a3601df0c8b196026aa5923c1b44dd2de6c187389f9fcaceb2a8de99952"}
Jul 16 21:52:38.690312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.690290    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:52:38.789179 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789148    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2hrt\" (UniqueName: \"kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789179 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789185    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789218    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789238    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789277    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789344    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789373    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config\") pod \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\" (UID: \"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1\") "
Jul 16 21:52:38.789833 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789785    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:38.789833 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789790    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca" (OuterVolumeSpecName: "service-ca") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:38.789833 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789820    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:38.790223 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.789945    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config" (OuterVolumeSpecName: "console-config") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:38.791523 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.791497    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:38.791616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.791518    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt" (OuterVolumeSpecName: "kube-api-access-l2hrt") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "kube-api-access-l2hrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:38.791616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.791571    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" (UID: "8c939d39-59f9-4f9b-b05c-f9d5a781a7e1"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.889967    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.889997    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-oauth-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.890006    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2hrt\" (UniqueName: \"kubernetes.io/projected/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-kube-api-access-l2hrt\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.890015    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-trusted-ca-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.890024    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-console-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.890033    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:38.890049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:38.890041    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1-oauth-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:39.645942 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.645906    2555 generic.go:356] "Generic (PLEG): container finished" podID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerID="8fe503c8053ea2d320d54b11ebb6c34cf96f3eb0ee97a109e0914ad619a6a2ea" exitCode=0
Jul 16 21:52:39.646343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.645991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd" event={"ID":"841a5ddb-bf51-4179-aafe-ebd6e76373c8","Type":"ContainerDied","Data":"8fe503c8053ea2d320d54b11ebb6c34cf96f3eb0ee97a109e0914ad619a6a2ea"}
Jul 16 21:52:39.647431 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.647396    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7569495f5c-hs8xn" event={"ID":"8c939d39-59f9-4f9b-b05c-f9d5a781a7e1","Type":"ContainerDied","Data":"a4e55fcd3a11268864e7c5c89dcd6b57d91b2c1393c44e97bbc82c1d55483954"}
Jul 16 21:52:39.647547 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.647449    2555 scope.go:117] "RemoveContainer" containerID="148c7a3601df0c8b196026aa5923c1b44dd2de6c187389f9fcaceb2a8de99952"
Jul 16 21:52:39.647547 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.647481    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7569495f5c-hs8xn"
Jul 16 21:52:39.676771 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.676747    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:52:39.680723 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:39.680704    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-7569495f5c-hs8xn"]
Jul 16 21:52:41.658225 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:41.658103    2555 generic.go:356] "Generic (PLEG): container finished" podID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerID="5845b7aa808859aa999cbb56ccb24e18e5c68543f831e46efccd62c0f1c915fd" exitCode=0
Jul 16 21:52:41.658225 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:41.658177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd" event={"ID":"841a5ddb-bf51-4179-aafe-ebd6e76373c8","Type":"ContainerDied","Data":"5845b7aa808859aa999cbb56ccb24e18e5c68543f831e46efccd62c0f1c915fd"}
Jul 16 21:52:41.675045 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:41.675016    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" path="/var/lib/kubelet/pods/8c939d39-59f9-4f9b-b05c-f9d5a781a7e1/volumes"
Jul 16 21:52:48.692416 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:48.692386    2555 generic.go:356] "Generic (PLEG): container finished" podID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerID="3d107e856bea6b155f62ae5b01acb307ebb069522217844cb3f38dff027cfae2" exitCode=0
Jul 16 21:52:48.692811 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:48.692478    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd" event={"ID":"841a5ddb-bf51-4179-aafe-ebd6e76373c8","Type":"ContainerDied","Data":"3d107e856bea6b155f62ae5b01acb307ebb069522217844cb3f38dff027cfae2"}
Jul 16 21:52:49.823693 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.823671    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:49.878130 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.878100    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util\") pod \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") "
Jul 16 21:52:49.878261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.878147    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7z8fz\" (UniqueName: \"kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz\") pod \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") "
Jul 16 21:52:49.878261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.878177    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle\") pod \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\" (UID: \"841a5ddb-bf51-4179-aafe-ebd6e76373c8\") "
Jul 16 21:52:49.878910 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.878840    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle" (OuterVolumeSpecName: "bundle") pod "841a5ddb-bf51-4179-aafe-ebd6e76373c8" (UID: "841a5ddb-bf51-4179-aafe-ebd6e76373c8"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:52:49.880433 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.880405    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz" (OuterVolumeSpecName: "kube-api-access-7z8fz") pod "841a5ddb-bf51-4179-aafe-ebd6e76373c8" (UID: "841a5ddb-bf51-4179-aafe-ebd6e76373c8"). InnerVolumeSpecName "kube-api-access-7z8fz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:49.882344 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.882325    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util" (OuterVolumeSpecName: "util") pod "841a5ddb-bf51-4179-aafe-ebd6e76373c8" (UID: "841a5ddb-bf51-4179-aafe-ebd6e76373c8"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:52:49.979637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.979557    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:49.979637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.979584    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/841a5ddb-bf51-4179-aafe-ebd6e76373c8-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:49.979637 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:49.979593    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7z8fz\" (UniqueName: \"kubernetes.io/projected/841a5ddb-bf51-4179-aafe-ebd6e76373c8-kube-api-access-7z8fz\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:52:50.702281 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:50.702253    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd"
Jul 16 21:52:50.702448 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:50.702252    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/685f1ed643dcce69727e2826bbf9be585096b4cf774051b5daa5e67540bh5fd" event={"ID":"841a5ddb-bf51-4179-aafe-ebd6e76373c8","Type":"ContainerDied","Data":"3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032"}
Jul 16 21:52:50.702448 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:50.702374    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3812a564015aec9a10da7bb57df4fd8a7c105c4dbb968b5d2f281a413ad55032"
Jul 16 21:52:56.492349 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492311    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"]
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492601    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="extract"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492612    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="extract"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492627    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="util"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492632    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="util"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492640    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="pull"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492645    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="pull"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492653    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerName="console"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492658    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerName="console"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492700    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c939d39-59f9-4f9b-b05c-f9d5a781a7e1" containerName="console"
Jul 16 21:52:56.492742 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.492707    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="841a5ddb-bf51-4179-aafe-ebd6e76373c8" containerName="extract"
Jul 16 21:52:56.494592 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.494573    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.497077 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.497053    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager-operator\"/\"cert-manager-operator-controller-manager-dockercfg-km226\""
Jul 16 21:52:56.497214 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.497194    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:52:56.497363 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.497349    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager-operator\"/\"kube-root-ca.crt\""
Jul 16 21:52:56.513735 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.513710    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"]
Jul 16 21:52:56.528656 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.528629    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srjbd\" (UniqueName: \"kubernetes.io/projected/812167da-2068-433e-8146-95a365c9a15c-kube-api-access-srjbd\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.528760 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.528673    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812167da-2068-433e-8146-95a365c9a15c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.630120 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.630088    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srjbd\" (UniqueName: \"kubernetes.io/projected/812167da-2068-433e-8146-95a365c9a15c-kube-api-access-srjbd\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.630120 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.630133    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812167da-2068-433e-8146-95a365c9a15c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.630488 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.630450    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/812167da-2068-433e-8146-95a365c9a15c-tmp\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.642248 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.642229    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srjbd\" (UniqueName: \"kubernetes.io/projected/812167da-2068-433e-8146-95a365c9a15c-kube-api-access-srjbd\") pod \"cert-manager-operator-controller-manager-55c444bf86-ngfhg\" (UID: \"812167da-2068-433e-8146-95a365c9a15c\") " pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.803647 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.803616    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"
Jul 16 21:52:56.955148 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:56.955116    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg"]
Jul 16 21:52:56.958427 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:52:56.958394    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod812167da_2068_433e_8146_95a365c9a15c.slice/crio-9765e85fa07d546ab9d3f6c737548a6f883cf9910894617d6a729e7a2c9d2622 WatchSource:0}: Error finding container 9765e85fa07d546ab9d3f6c737548a6f883cf9910894617d6a729e7a2c9d2622: Status 404 returned error can't find the container with id 9765e85fa07d546ab9d3f6c737548a6f883cf9910894617d6a729e7a2c9d2622
Jul 16 21:52:57.732139 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:57.732100    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg" event={"ID":"812167da-2068-433e-8146-95a365c9a15c","Type":"ContainerStarted","Data":"9765e85fa07d546ab9d3f6c737548a6f883cf9910894617d6a729e7a2c9d2622"}
Jul 16 21:52:58.739013 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:58.738917    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg" event={"ID":"812167da-2068-433e-8146-95a365c9a15c","Type":"ContainerStarted","Data":"cd33c437f093fb937718269b792a66cda4bb366104c5dc73e50c965d8ef63cc5"}
Jul 16 21:52:58.767930 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:52:58.767879    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager-operator/cert-manager-operator-controller-manager-55c444bf86-ngfhg" podStartSLOduration=1.363386518 podStartE2EDuration="2.767865149s" totalImagesPullingTime="1.404478631s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:52:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:58.765867534 +0000 UTC m=+233.602921347" watchObservedRunningTime="2026-07-16 21:52:58.767865149 +0000 UTC m=+233.604918948"
Jul 16 21:53:00.142976 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.142946    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"]
Jul 16 21:53:00.145267 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.145247    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.148855 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.148831    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:53:00.149047 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.149031    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:53:00.150927 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.150907    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggr5m\""
Jul 16 21:53:00.166901 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.166874    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"]
Jul 16 21:53:00.264668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.264631    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.264827 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.264678    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.264827 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.264747    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zg7b\" (UniqueName: \"kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.366025 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.365985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.366180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.366037    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.366180 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.366078    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zg7b\" (UniqueName: \"kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.366515 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.366488    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.366556 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.366503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.387286 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.387254    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zg7b\" (UniqueName: \"kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b\") pod \"77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") " pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.454144 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.454058    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:00.595103 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.595075    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"]
Jul 16 21:53:00.597423 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:00.597394    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24074324_7003_4aec_b398_7674e89f2d27.slice/crio-d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0 WatchSource:0}: Error finding container d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0: Status 404 returned error can't find the container with id d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0
Jul 16 21:53:00.749315 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.749227    2555 generic.go:356] "Generic (PLEG): container finished" podID="24074324-7003-4aec-b398-7674e89f2d27" containerID="e3e9b8923bfa9ebfb8abf7625579bb7d5dda2fc95a9d88fc224a0f41353ae433" exitCode=0
Jul 16 21:53:00.749315 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.749273    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws" event={"ID":"24074324-7003-4aec-b398-7674e89f2d27","Type":"ContainerDied","Data":"e3e9b8923bfa9ebfb8abf7625579bb7d5dda2fc95a9d88fc224a0f41353ae433"}
Jul 16 21:53:00.749507 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:00.749321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws" event={"ID":"24074324-7003-4aec-b398-7674e89f2d27","Type":"ContainerStarted","Data":"d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0"}
Jul 16 21:53:02.760544 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:02.760514    2555 generic.go:356] "Generic (PLEG): container finished" podID="24074324-7003-4aec-b398-7674e89f2d27" containerID="0138c7e272a6a6ebb79d968a7931b8c774df02151c9d6968814b450962ec90fa" exitCode=0
Jul 16 21:53:02.760911 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:02.760610    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws" event={"ID":"24074324-7003-4aec-b398-7674e89f2d27","Type":"ContainerDied","Data":"0138c7e272a6a6ebb79d968a7931b8c774df02151c9d6968814b450962ec90fa"}
Jul 16 21:53:03.767235 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:03.767196    2555 generic.go:356] "Generic (PLEG): container finished" podID="24074324-7003-4aec-b398-7674e89f2d27" containerID="ddba3540f354cadb19b23972d5862385bb8902c411e7210413120b02cdbfca11" exitCode=0
Jul 16 21:53:03.767640 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:03.767277    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws" event={"ID":"24074324-7003-4aec-b398-7674e89f2d27","Type":"ContainerDied","Data":"ddba3540f354cadb19b23972d5862385bb8902c411e7210413120b02cdbfca11"}
Jul 16 21:53:04.900028 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:04.900006    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:05.005396 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.005359    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle\") pod \"24074324-7003-4aec-b398-7674e89f2d27\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") "
Jul 16 21:53:05.005616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.005428    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util\") pod \"24074324-7003-4aec-b398-7674e89f2d27\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") "
Jul 16 21:53:05.005616 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.005548    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7zg7b\" (UniqueName: \"kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b\") pod \"24074324-7003-4aec-b398-7674e89f2d27\" (UID: \"24074324-7003-4aec-b398-7674e89f2d27\") "
Jul 16 21:53:05.005885 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.005851    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle" (OuterVolumeSpecName: "bundle") pod "24074324-7003-4aec-b398-7674e89f2d27" (UID: "24074324-7003-4aec-b398-7674e89f2d27"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:53:05.007708 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.007680    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b" (OuterVolumeSpecName: "kube-api-access-7zg7b") pod "24074324-7003-4aec-b398-7674e89f2d27" (UID: "24074324-7003-4aec-b398-7674e89f2d27"). InnerVolumeSpecName "kube-api-access-7zg7b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:53:05.010999 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.010979    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util" (OuterVolumeSpecName: "util") pod "24074324-7003-4aec-b398-7674e89f2d27" (UID: "24074324-7003-4aec-b398-7674e89f2d27"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:53:05.106246 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.106200    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zg7b\" (UniqueName: \"kubernetes.io/projected/24074324-7003-4aec-b398-7674e89f2d27-kube-api-access-7zg7b\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:05.106246 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.106244    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:05.106246 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.106254    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/24074324-7003-4aec-b398-7674e89f2d27-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:05.776987 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.776965    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws"
Jul 16 21:53:05.776987 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.776973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/77defbb6647882b321b46d16bdefe62633c3425354d3d93571a1d6a87fczbws" event={"ID":"24074324-7003-4aec-b398-7674e89f2d27","Type":"ContainerDied","Data":"d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0"}
Jul 16 21:53:05.777175 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:05.777016    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5df98888eab014878de04212cd26f0b2b0ac701c0c8aeaa914ee5eab5a0baa0"
Jul 16 21:53:06.258610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258576    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"]
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258865    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="pull"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258875    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="pull"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258896    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="util"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258901    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="util"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258911    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="extract"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258916    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="extract"
Jul 16 21:53:06.258990 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.258962    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="24074324-7003-4aec-b398-7674e89f2d27" containerName="extract"
Jul 16 21:53:06.261829 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.261813    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.264638 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.264608    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Jul 16 21:53:06.264769 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.264749    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Jul 16 21:53:06.265619 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.265602    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6x62l\""
Jul 16 21:53:06.271057 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.271038    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"]
Jul 16 21:53:06.317488 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.317442    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.317651 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.317506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnwwx\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-kube-api-access-cnwwx\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.418839 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.418803    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.418977 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.418861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnwwx\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-kube-api-access-cnwwx\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.427187 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.427159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-bound-sa-token\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.427299 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.427218    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnwwx\" (UniqueName: \"kubernetes.io/projected/8b1e51ed-aa63-49d1-adef-968f912f16b7-kube-api-access-cnwwx\") pod \"cert-manager-cainjector-559cdd8c87-9k25h\" (UID: \"8b1e51ed-aa63-49d1-adef-968f912f16b7\") " pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.584750 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.584706    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"
Jul 16 21:53:06.711266 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.711230    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-559cdd8c87-9k25h"]
Jul 16 21:53:06.714174 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:06.714148    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8b1e51ed_aa63_49d1_adef_968f912f16b7.slice/crio-9208eaa199cd9f64c78043ae74cf8f3ac0c4e1f0c9035196c10419760d9b106e WatchSource:0}: Error finding container 9208eaa199cd9f64c78043ae74cf8f3ac0c4e1f0c9035196c10419760d9b106e: Status 404 returned error can't find the container with id 9208eaa199cd9f64c78043ae74cf8f3ac0c4e1f0c9035196c10419760d9b106e
Jul 16 21:53:06.782725 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:06.782688    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h" event={"ID":"8b1e51ed-aa63-49d1-adef-968f912f16b7","Type":"ContainerStarted","Data":"9208eaa199cd9f64c78043ae74cf8f3ac0c4e1f0c9035196c10419760d9b106e"}
Jul 16 21:53:09.797807 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:09.797769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h" event={"ID":"8b1e51ed-aa63-49d1-adef-968f912f16b7","Type":"ContainerStarted","Data":"88bce9b6d2cbc9db6619ba9c9a4444acb84516998f6281370cd24d5db91f8ad6"}
Jul 16 21:53:10.340767 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.340713    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-559cdd8c87-9k25h" podStartSLOduration=1.838541032 podStartE2EDuration="4.340692966s" totalImagesPullingTime="2.502151934s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:09.812817995 +0000 UTC m=+244.649871796" watchObservedRunningTime="2026-07-16 21:53:10.340692966 +0000 UTC m=+245.177746798"
Jul 16 21:53:10.341207 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.341192    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"]
Jul 16 21:53:10.344237 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.344222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.347587 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.347559    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-v4fkl\""
Jul 16 21:53:10.347698 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.347621    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:53:10.347698 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.347666    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 16 21:53:10.358032 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.358009    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"]
Jul 16 21:53:10.450164 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.450130    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.450348 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.450195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9td2\" (UniqueName: \"kubernetes.io/projected/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-kube-api-access-m9td2\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.550820 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.550766    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9td2\" (UniqueName: \"kubernetes.io/projected/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-kube-api-access-m9td2\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.551004 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.550855    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.551236 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.551215    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-tmp\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.560204 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.560177    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9td2\" (UniqueName: \"kubernetes.io/projected/c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098-kube-api-access-m9td2\") pod \"openshift-lws-operator-fd8ccff4c-8zc4c\" (UID: \"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.653302 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.653228    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"
Jul 16 21:53:10.786277 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.786250    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c"]
Jul 16 21:53:10.788711 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:10.788679    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc8f7e5ff_13b7_41da_9e1b_f7d1b0bb6098.slice/crio-4d6356f73fed9debbee82df457111f63b76c0bfa69ad2c49dcbc7270cf544ad4 WatchSource:0}: Error finding container 4d6356f73fed9debbee82df457111f63b76c0bfa69ad2c49dcbc7270cf544ad4: Status 404 returned error can't find the container with id 4d6356f73fed9debbee82df457111f63b76c0bfa69ad2c49dcbc7270cf544ad4
Jul 16 21:53:10.802940 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:10.802911    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c" event={"ID":"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098","Type":"ContainerStarted","Data":"4d6356f73fed9debbee82df457111f63b76c0bfa69ad2c49dcbc7270cf544ad4"}
Jul 16 21:53:12.270179 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.270138    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5bnr"]
Jul 16 21:53:12.285069 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.285038    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.289365 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.289288    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-t5b4z\""
Jul 16 21:53:12.292343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.291960    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5bnr"]
Jul 16 21:53:12.367864 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.367824    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.368056 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.367884    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnn75\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-kube-api-access-xnn75\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.469349 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.469311    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.469564 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.469365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnn75\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-kube-api-access-xnn75\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.478584 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.478547    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-bound-sa-token\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.478732 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.478627    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnn75\" (UniqueName: \"kubernetes.io/projected/5f49f5f0-b28e-4b46-95d6-f6742349736a-kube-api-access-xnn75\") pod \"cert-manager-webhook-6486ffd997-k5bnr\" (UID: \"5f49f5f0-b28e-4b46-95d6-f6742349736a\") " pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.600960 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.600924    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:12.851545 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:12.851523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-6486ffd997-k5bnr"]
Jul 16 21:53:12.853410 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:12.853386    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f49f5f0_b28e_4b46_95d6_f6742349736a.slice/crio-34d693b660e3b4632a016cde42318b13b269b2e15010ad21d2485346e7c4b532 WatchSource:0}: Error finding container 34d693b660e3b4632a016cde42318b13b269b2e15010ad21d2485346e7c4b532: Status 404 returned error can't find the container with id 34d693b660e3b4632a016cde42318b13b269b2e15010ad21d2485346e7c4b532
Jul 16 21:53:13.819625 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.819585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c" event={"ID":"c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098","Type":"ContainerStarted","Data":"f1232b34617cf590097f75233be8679c6253df639b79c5884acbf4347f278d05"}
Jul 16 21:53:13.821065 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.821042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr" event={"ID":"5f49f5f0-b28e-4b46-95d6-f6742349736a","Type":"ContainerStarted","Data":"0cebadfaa2ff08265981b56601da58bed8e781cdf739f71ffcc144b9fa8362f9"}
Jul 16 21:53:13.821065 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.821069    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr" event={"ID":"5f49f5f0-b28e-4b46-95d6-f6742349736a","Type":"ContainerStarted","Data":"34d693b660e3b4632a016cde42318b13b269b2e15010ad21d2485346e7c4b532"}
Jul 16 21:53:13.821209 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.821153    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:13.840732 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.840683    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-8zc4c" podStartSLOduration=1.8522614339999999 podStartE2EDuration="3.840669773s" totalImagesPullingTime="1.988408339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:13.839843257 +0000 UTC m=+248.676897057" watchObservedRunningTime="2026-07-16 21:53:13.840669773 +0000 UTC m=+248.677723572"
Jul 16 21:53:13.857111 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:13.857073    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr" podStartSLOduration=1.8570575599999999 podStartE2EDuration="1.85705756s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:13.855601233 +0000 UTC m=+248.692655043" watchObservedRunningTime="2026-07-16 21:53:13.85705756 +0000 UTC m=+248.694111362"
Jul 16 21:53:16.791242 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.791206    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"]
Jul 16 21:53:16.793745 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.793726    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:16.796540 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.796517    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:53:16.797065 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.797048    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:53:16.797364 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.797351    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-ggr5m\""
Jul 16 21:53:16.805540 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.805518    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"]
Jul 16 21:53:16.904305 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.904266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtx74\" (UniqueName: \"kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:16.904486 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.904334    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:16.904486 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:16.904393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.004796 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.004762    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtx74\" (UniqueName: \"kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.004965 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.004813    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.004965 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.004865    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.005333 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.005311    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.005367 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.005323    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.013988 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.013961    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtx74\" (UniqueName: \"kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74\") pod \"629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") " pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.103033 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.102945    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:17.234679 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.234655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"]
Jul 16 21:53:17.236927 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:17.236899    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ccaa129_581c_4042_bc78_43ce2864eb62.slice/crio-1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9 WatchSource:0}: Error finding container 1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9: Status 404 returned error can't find the container with id 1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9
Jul 16 21:53:17.840387 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.840346    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerID="edf119f71ab1272b0f9fbf18f42f79d61e981941841a343e9d1d331a902e6a51" exitCode=0
Jul 16 21:53:17.840870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.840436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd" event={"ID":"7ccaa129-581c-4042-bc78-43ce2864eb62","Type":"ContainerDied","Data":"edf119f71ab1272b0f9fbf18f42f79d61e981941841a343e9d1d331a902e6a51"}
Jul 16 21:53:17.840870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:17.840509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd" event={"ID":"7ccaa129-581c-4042-bc78-43ce2864eb62","Type":"ContainerStarted","Data":"1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9"}
Jul 16 21:53:19.828031 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:19.828000    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-6486ffd997-k5bnr"
Jul 16 21:53:19.851292 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:19.851256    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerID="f47e106bd5a773ca61eefcd3df6fec8f542827c62c1b85b8655718d987939531" exitCode=0
Jul 16 21:53:19.851485 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:19.851289    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd" event={"ID":"7ccaa129-581c-4042-bc78-43ce2864eb62","Type":"ContainerDied","Data":"f47e106bd5a773ca61eefcd3df6fec8f542827c62c1b85b8655718d987939531"}
Jul 16 21:53:20.477910 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.477833    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fs6d4"]
Jul 16 21:53:20.488587 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.488566    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.491312 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.491283    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-f2nxv\""
Jul 16 21:53:20.492684 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.492663    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fs6d4"]
Jul 16 21:53:20.637489 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.637429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-584zz\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-kube-api-access-584zz\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.637663 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.637518    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-bound-sa-token\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.738726 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.738643    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-bound-sa-token\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.738726 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.738723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-584zz\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-kube-api-access-584zz\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.757434 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.757401    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-bound-sa-token\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.757577 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.757536    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-584zz\" (UniqueName: \"kubernetes.io/projected/88e3a48c-bc6b-4678-8bd3-31c184b64c63-kube-api-access-584zz\") pod \"cert-manager-55b4d954d5-fs6d4\" (UID: \"88e3a48c-bc6b-4678-8bd3-31c184b64c63\") " pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.798428 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.798393    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-fs6d4"
Jul 16 21:53:20.858407 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.858373    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerID="4b364aefb52a21c65da0dbc4ff3058071d56e38470cfb940d193a3bd62080d4b" exitCode=0
Jul 16 21:53:20.858407 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.858423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd" event={"ID":"7ccaa129-581c-4042-bc78-43ce2864eb62","Type":"ContainerDied","Data":"4b364aefb52a21c65da0dbc4ff3058071d56e38470cfb940d193a3bd62080d4b"}
Jul 16 21:53:20.949358 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:20.949337    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-fs6d4"]
Jul 16 21:53:20.951752 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:20.951724    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e3a48c_bc6b_4678_8bd3_31c184b64c63.slice/crio-7741fc10c054b3ece13f870ea3913f2b3555647a7e0928f9527210a1b1f75949 WatchSource:0}: Error finding container 7741fc10c054b3ece13f870ea3913f2b3555647a7e0928f9527210a1b1f75949: Status 404 returned error can't find the container with id 7741fc10c054b3ece13f870ea3913f2b3555647a7e0928f9527210a1b1f75949
Jul 16 21:53:21.864367 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:21.864329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fs6d4" event={"ID":"88e3a48c-bc6b-4678-8bd3-31c184b64c63","Type":"ContainerStarted","Data":"665429f4001f7e71b88dcfd5a18a5c55cfc6ccf1b15f70a35b9120944b37d8fb"}
Jul 16 21:53:21.864367 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:21.864370    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-fs6d4" event={"ID":"88e3a48c-bc6b-4678-8bd3-31c184b64c63","Type":"ContainerStarted","Data":"7741fc10c054b3ece13f870ea3913f2b3555647a7e0928f9527210a1b1f75949"}
Jul 16 21:53:21.881456 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:21.881380    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-fs6d4" podStartSLOduration=1.881364816 podStartE2EDuration="1.881364816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:21.880722954 +0000 UTC m=+256.717776755" watchObservedRunningTime="2026-07-16 21:53:21.881364816 +0000 UTC m=+256.718418616"
Jul 16 21:53:21.994106 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:21.994081    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:22.150519 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.150396    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle\") pod \"7ccaa129-581c-4042-bc78-43ce2864eb62\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") "
Jul 16 21:53:22.150668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.150523    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util\") pod \"7ccaa129-581c-4042-bc78-43ce2864eb62\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") "
Jul 16 21:53:22.150668 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.150558    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtx74\" (UniqueName: \"kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74\") pod \"7ccaa129-581c-4042-bc78-43ce2864eb62\" (UID: \"7ccaa129-581c-4042-bc78-43ce2864eb62\") "
Jul 16 21:53:22.151121 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.151099    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle" (OuterVolumeSpecName: "bundle") pod "7ccaa129-581c-4042-bc78-43ce2864eb62" (UID: "7ccaa129-581c-4042-bc78-43ce2864eb62"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:53:22.152906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.152875    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74" (OuterVolumeSpecName: "kube-api-access-vtx74") pod "7ccaa129-581c-4042-bc78-43ce2864eb62" (UID: "7ccaa129-581c-4042-bc78-43ce2864eb62"). InnerVolumeSpecName "kube-api-access-vtx74". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:53:22.155961 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.155933    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util" (OuterVolumeSpecName: "util") pod "7ccaa129-581c-4042-bc78-43ce2864eb62" (UID: "7ccaa129-581c-4042-bc78-43ce2864eb62"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:53:22.251604 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.251559    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:22.251604 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.251596    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/7ccaa129-581c-4042-bc78-43ce2864eb62-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:22.251604 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.251606    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtx74\" (UniqueName: \"kubernetes.io/projected/7ccaa129-581c-4042-bc78-43ce2864eb62-kube-api-access-vtx74\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:53:22.870944 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.870914    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd"
Jul 16 21:53:22.871340 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.870945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/629fd74b02459b8d9ee224b8bdbb513172290d80b07a5bf327840796af7lmrd" event={"ID":"7ccaa129-581c-4042-bc78-43ce2864eb62","Type":"ContainerDied","Data":"1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9"}
Jul 16 21:53:22.871340 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:22.870990    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1aa3e47f6c197ff9cc1d904b0cc5c17eac98b8df028fd7d25991074e79caa0a9"
Jul 16 21:53:34.169687 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169609    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"]
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169933    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="extract"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169945    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="extract"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169962    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="util"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="util"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.169974    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="pull"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.170008    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="pull"
Jul 16 21:53:34.170096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.170079    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ccaa129-581c-4042-bc78-43ce2864eb62" containerName="extract"
Jul 16 21:53:34.172283 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.172262    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.176970 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.176950    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 16 21:53:34.177093 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.177041    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 16 21:53:34.177655 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.177625    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-hk7b8\""
Jul 16 21:53:34.177838 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.177820    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 16 21:53:34.178331 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.178314    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 16 21:53:34.197135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.197107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"]
Jul 16 21:53:34.355017 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.354986    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-manifests\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.355017 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.355021    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-tmp\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.355228 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.355048    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.355228 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.355077    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.355228 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.355161    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xzbws\" (UniqueName: \"kubernetes.io/projected/848b39a2-b37d-4a14-bfb8-b250294e70bf-kube-api-access-xzbws\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.456419 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.456326    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.456419 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.456377    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xzbws\" (UniqueName: \"kubernetes.io/projected/848b39a2-b37d-4a14-bfb8-b250294e70bf-kube-api-access-xzbws\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.456645 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.456430    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-manifests\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.456645 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.456451    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-tmp\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.456739 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.456665    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.457073 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.457013    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-manifests\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.458801 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.458775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/848b39a2-b37d-4a14-bfb8-b250294e70bf-tmp\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.458941 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.458923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-webhook-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.459106 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.459088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/848b39a2-b37d-4a14-bfb8-b250294e70bf-apiservice-cert\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.482813 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.482778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xzbws\" (UniqueName: \"kubernetes.io/projected/848b39a2-b37d-4a14-bfb8-b250294e70bf-kube-api-access-xzbws\") pod \"opendatahub-operator-controller-manager-54cf7dfc9c-zstcg\" (UID: \"848b39a2-b37d-4a14-bfb8-b250294e70bf\") " pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.782412 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.782380    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:34.934353 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:34.934323    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"]
Jul 16 21:53:34.937658 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:34.937628    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod848b39a2_b37d_4a14_bfb8_b250294e70bf.slice/crio-933d5504b2ee39671498aa4556e0703395108568ae042f6722c3daa736b02b57 WatchSource:0}: Error finding container 933d5504b2ee39671498aa4556e0703395108568ae042f6722c3daa736b02b57: Status 404 returned error can't find the container with id 933d5504b2ee39671498aa4556e0703395108568ae042f6722c3daa736b02b57
Jul 16 21:53:35.932922 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:35.932882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg" event={"ID":"848b39a2-b37d-4a14-bfb8-b250294e70bf","Type":"ContainerStarted","Data":"933d5504b2ee39671498aa4556e0703395108568ae042f6722c3daa736b02b57"}
Jul 16 21:53:37.943025 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:37.942990    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg" event={"ID":"848b39a2-b37d-4a14-bfb8-b250294e70bf","Type":"ContainerStarted","Data":"199a20386fd7c2d40f58f091bdf11499ae4e69dd529e7cdf804f00deea580922"}
Jul 16 21:53:38.948565 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:38.948532    2555 generic.go:356] "Generic (PLEG): container finished" podID="848b39a2-b37d-4a14-bfb8-b250294e70bf" containerID="199a20386fd7c2d40f58f091bdf11499ae4e69dd529e7cdf804f00deea580922" exitCode=0
Jul 16 21:53:38.948933 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:38.948615    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg" event={"ID":"848b39a2-b37d-4a14-bfb8-b250294e70bf","Type":"ContainerDied","Data":"199a20386fd7c2d40f58f091bdf11499ae4e69dd529e7cdf804f00deea580922"}
Jul 16 21:53:39.954704 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:39.954672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg" event={"ID":"848b39a2-b37d-4a14-bfb8-b250294e70bf","Type":"ContainerStarted","Data":"90036c5df480661bdf15e98e6416488abc0956dc2d745b07cc9f27f7ee1038dc"}
Jul 16 21:53:39.955080 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:39.954775    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:50.961692 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:50.961656    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg"
Jul 16 21:53:50.996404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:50.996349    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-54cf7dfc9c-zstcg" podStartSLOduration=12.727059682 podStartE2EDuration="16.996324959s" totalImagesPullingTime="3.247114447s" totalInitContainerRuntime="1.02215083s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:34 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:40.019062198 +0000 UTC m=+274.856116001" watchObservedRunningTime="2026-07-16 21:53:50.996324959 +0000 UTC m=+285.833378759"
Jul 16 21:53:58.005396 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.005343    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"]
Jul 16 21:53:58.009873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.009846    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.013982 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.013957    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 16 21:53:58.014286 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.014262    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 16 21:53:58.014555 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.014531    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 16 21:53:58.014782 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.014746    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rd79w\""
Jul 16 21:53:58.029081 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.029051    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"]
Jul 16 21:53:58.046243 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.046210    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-metrics-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.046407 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.046266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4lz5\" (UniqueName: \"kubernetes.io/projected/a79e15e6-e1c5-410b-b171-535033d810ea-kube-api-access-l4lz5\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.046407 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.046358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a79e15e6-e1c5-410b-b171-535033d810ea-manager-config\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.046558 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.046429    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.147728 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.147691    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4lz5\" (UniqueName: \"kubernetes.io/projected/a79e15e6-e1c5-410b-b171-535033d810ea-kube-api-access-l4lz5\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.147920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.147744    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a79e15e6-e1c5-410b-b171-535033d810ea-manager-config\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.147920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.147775    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.147920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.147867    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-metrics-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.148597 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.148567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a79e15e6-e1c5-410b-b171-535033d810ea-manager-config\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.150420 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.150395    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.150581 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.150564    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/a79e15e6-e1c5-410b-b171-535033d810ea-metrics-cert\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.158203 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.158184    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4lz5\" (UniqueName: \"kubernetes.io/projected/a79e15e6-e1c5-410b-b171-535033d810ea-kube-api-access-l4lz5\") pod \"lws-controller-manager-656bfb4b98-6gjq4\" (UID: \"a79e15e6-e1c5-410b-b171-535033d810ea\") " pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.321647 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.321611    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:53:58.459433 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:58.459361    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"]
Jul 16 21:53:58.461921 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:53:58.461893    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda79e15e6_e1c5_410b_b171_535033d810ea.slice/crio-55e15a63e66ff94a38fb556ae3e11f963f62d9795c2dba452363e472248aa5c4 WatchSource:0}: Error finding container 55e15a63e66ff94a38fb556ae3e11f963f62d9795c2dba452363e472248aa5c4: Status 404 returned error can't find the container with id 55e15a63e66ff94a38fb556ae3e11f963f62d9795c2dba452363e472248aa5c4
Jul 16 21:53:59.046360 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:53:59.046322    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4" event={"ID":"a79e15e6-e1c5-410b-b171-535033d810ea","Type":"ContainerStarted","Data":"55e15a63e66ff94a38fb556ae3e11f963f62d9795c2dba452363e472248aa5c4"}
Jul 16 21:54:01.058807 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:01.058775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4" event={"ID":"a79e15e6-e1c5-410b-b171-535033d810ea","Type":"ContainerStarted","Data":"5b7e01b9e89f7437af688b371dbf73311adffa52cd5a28a35279c6b30349393e"}
Jul 16 21:54:01.059256 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:01.058914    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:54:05.261709 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.261653    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4" podStartSLOduration=6.704367578 podStartE2EDuration="8.261636604s" totalImagesPullingTime="1.557269026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:01.104602769 +0000 UTC m=+295.941656655" watchObservedRunningTime="2026-07-16 21:54:05.261636604 +0000 UTC m=+300.098690435"
Jul 16 21:54:05.262610 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.262593    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"]
Jul 16 21:54:05.266121 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.266106    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.275135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.275113    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 16 21:54:05.275135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.275126    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 16 21:54:05.275306 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.275142    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 16 21:54:05.276242 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.276229    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zqvqp\""
Jul 16 21:54:05.276485 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.276450    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 16 21:54:05.276541 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.276515    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 16 21:54:05.277972 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.277956    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 16 21:54:05.315728 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.315728 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.315944 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315764    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.315944 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315820    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76185561-cce1-4dbe-b589-848a085a0df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.315944 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315862    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxlcx\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-kube-api-access-xxlcx\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.315944 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315882    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.316079 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.315934    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.323429 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.323399    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"]
Jul 16 21:54:05.416965 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.416929    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.416971    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417045    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417094    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76185561-cce1-4dbe-b589-848a085a0df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417143 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417137    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxlcx\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-kube-api-access-xxlcx\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417392 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417165    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.417775 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.417703    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.419670 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.419637    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/76185561-cce1-4dbe-b589-848a085a0df6-local-certs\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.419862 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.419830    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.419924 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.419846    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-cacerts\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.419982 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.419931    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/76185561-cce1-4dbe-b589-848a085a0df6-istio-kubeconfig\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.435646 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.435617    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-istio-token\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.454115 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.454082    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxlcx\" (UniqueName: \"kubernetes.io/projected/76185561-cce1-4dbe-b589-848a085a0df6-kube-api-access-xxlcx\") pod \"istiod-openshift-gateway-94bb8fbfd-94knq\" (UID: \"76185561-cce1-4dbe-b589-848a085a0df6\") " pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.562511 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.562486    2555 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:54:05.578920 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.578886    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-zqvqp\""
Jul 16 21:54:05.587040 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.587012    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:05.777270 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:05.777247    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"]
Jul 16 21:54:05.779087 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:54:05.779054    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76185561_cce1_4dbe_b589_848a085a0df6.slice/crio-80958e1b7aef4ef52660f583b1f9a12113613c6ea8c38295a50a943ebe4f6590 WatchSource:0}: Error finding container 80958e1b7aef4ef52660f583b1f9a12113613c6ea8c38295a50a943ebe4f6590: Status 404 returned error can't find the container with id 80958e1b7aef4ef52660f583b1f9a12113613c6ea8c38295a50a943ebe4f6590
Jul 16 21:54:06.084047 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:06.084001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq" event={"ID":"76185561-cce1-4dbe-b589-848a085a0df6","Type":"ContainerStarted","Data":"80958e1b7aef4ef52660f583b1f9a12113613c6ea8c38295a50a943ebe4f6590"}
Jul 16 21:54:08.110869 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:08.110833    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:54:08.111242 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:08.110910    2555 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892164Ki","pods":"250"}
Jul 16 21:54:09.101678 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:09.101644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq" event={"ID":"76185561-cce1-4dbe-b589-848a085a0df6","Type":"ContainerStarted","Data":"35c88fbe871b512acda6379b85bbd4e6f288b8f830b67ae727a1006a92d27c5b"}
Jul 16 21:54:09.101921 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:09.101865    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:09.103521 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:09.103496    2555 patch_prober.go:28] interesting pod/istiod-openshift-gateway-94bb8fbfd-94knq container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 16 21:54:09.103719 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:09.103686    2555 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq" podUID="76185561-cce1-4dbe-b589-848a085a0df6" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:54:09.132516 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:09.132444    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq" podStartSLOduration=1.8028505419999998 podStartE2EDuration="4.132421877s" totalImagesPullingTime="2.329571335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:09.130960914 +0000 UTC m=+303.968014707" watchObservedRunningTime="2026-07-16 21:54:09.132421877 +0000 UTC m=+303.969475677"
Jul 16 21:54:10.107489 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:10.107438    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-94bb8fbfd-94knq"
Jul 16 21:54:12.065670 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:54:12.065642    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-656bfb4b98-6gjq4"
Jul 16 21:55:10.454999 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.454909    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"]
Jul 16 21:55:10.458802 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.458779    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.461660 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.461640    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cbks6\""
Jul 16 21:55:10.461903 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.461888    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:55:10.462473 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.462440    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:55:10.473095 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.473071    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"]
Jul 16 21:55:10.583298 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.583266    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.583298 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.583304    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.583537 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.583326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77b6v\" (UniqueName: \"kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.684113 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.684078    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.684113 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.684117    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.684354 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.684140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77b6v\" (UniqueName: \"kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.684586 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.684564    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.684645 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.684573    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.694083 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.694048    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77b6v\" (UniqueName: \"kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v\") pod \"19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") " pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.768252 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.768181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:10.903832 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:10.903805    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"]
Jul 16 21:55:10.904490 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:55:10.904437    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode932528d_0c36_4366_828e_7b9a029bdfab.slice/crio-a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d WatchSource:0}: Error finding container a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d: Status 404 returned error can't find the container with id a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d
Jul 16 21:55:11.053794 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.053759    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"]
Jul 16 21:55:11.057504 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.057484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.068366 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.068343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"]
Jul 16 21:55:11.188500 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.188445    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.188500 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.188503    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xz2gf\" (UniqueName: \"kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.188715 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.188561    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.289349 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.289323    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xz2gf\" (UniqueName: \"kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.289532 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.289366    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.289532 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.289427    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.289827 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.289804    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.289864 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.289835    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.299261 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.299242    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xz2gf\" (UniqueName: \"kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf\") pod \"0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") " pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.366220 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.366162    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:11.391345 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.391311    2555 generic.go:356] "Generic (PLEG): container finished" podID="e932528d-0c36-4366-828e-7b9a029bdfab" containerID="55b01c28f4a77d7ceb3c8b73b5378c048321b3e78b53f34f771d104b59deb67e" exitCode=0
Jul 16 21:55:11.391503 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.391352    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm" event={"ID":"e932528d-0c36-4366-828e-7b9a029bdfab","Type":"ContainerDied","Data":"55b01c28f4a77d7ceb3c8b73b5378c048321b3e78b53f34f771d104b59deb67e"}
Jul 16 21:55:11.391503 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.391407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm" event={"ID":"e932528d-0c36-4366-828e-7b9a029bdfab","Type":"ContainerStarted","Data":"a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d"}
Jul 16 21:55:11.391880 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.391864    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:55:11.435995 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.435901    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"]
Jul 16 21:55:11.439909 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.439887    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.448362 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.448340    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"]
Jul 16 21:55:11.506455 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.506425    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"]
Jul 16 21:55:11.509891 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:55:11.509865    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91169da_c866_4383_b08f_2dc5b7d52914.slice/crio-eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3 WatchSource:0}: Error finding container eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3: Status 404 returned error can't find the container with id eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3
Jul 16 21:55:11.591870 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.591837    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.592002 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.591888    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt98m\" (UniqueName: \"kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.592064 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.592031    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.692883 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.692799    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.692883 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.692856    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.693074 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.692984    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt98m\" (UniqueName: \"kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.693174 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.693159    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.693232 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.693197    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.700934 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.700910    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt98m\" (UniqueName: \"kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m\") pod \"9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") " pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.750744 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.750711    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:11.836933 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.836833    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"]
Jul 16 21:55:11.840956 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.840928    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:11.851085 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.850842    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"]
Jul 16 21:55:11.895731 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.895707    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"]
Jul 16 21:55:11.898182 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:55:11.898153    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbcdc05_f3f0_4335_bc0a_57fc8b55ec13.slice/crio-d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f WatchSource:0}: Error finding container d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f: Status 404 returned error can't find the container with id d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f
Jul 16 21:55:11.996712 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.996683    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgjnh\" (UniqueName: \"kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:11.996860 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.996723    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:11.996860 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:11.996775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.097320 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.097297    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgjnh\" (UniqueName: \"kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.097413 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.097328    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.097413 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.097367    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.097772 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.097748    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.097856 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.097789    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.108034 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.108011    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgjnh\" (UniqueName: \"kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh\") pod \"5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") " pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.154798 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.154778    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:12.291327 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.291303    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"]
Jul 16 21:55:12.293060 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:55:12.293034    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02182b5c_0f36_4b77_945c_cc1c8c36305c.slice/crio-98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02 WatchSource:0}: Error finding container 98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02: Status 404 returned error can't find the container with id 98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02
Jul 16 21:55:12.398221 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.398192    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerID="d50fef880ff29c4902351659e2c28d481a1165e426438ac93865d90a69d352c7" exitCode=0
Jul 16 21:55:12.398364 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.398282    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7" event={"ID":"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13","Type":"ContainerDied","Data":"d50fef880ff29c4902351659e2c28d481a1165e426438ac93865d90a69d352c7"}
Jul 16 21:55:12.398364 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.398340    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7" event={"ID":"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13","Type":"ContainerStarted","Data":"d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f"}
Jul 16 21:55:12.400744 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.400718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerStarted","Data":"9dc1dd34a27e4578cd13a1b4344d45162f0d4f1421d81a89b648e95a2fb0498f"}
Jul 16 21:55:12.400847 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.400768    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerStarted","Data":"98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02"}
Jul 16 21:55:12.402384 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.402363    2555 generic.go:356] "Generic (PLEG): container finished" podID="b91169da-c866-4383-b08f-2dc5b7d52914" containerID="ea82982eb46752050fa93e3096909f09fee6d29c16aaba81655357b1c385441c" exitCode=0
Jul 16 21:55:12.402450 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.402436    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278" event={"ID":"b91169da-c866-4383-b08f-2dc5b7d52914","Type":"ContainerDied","Data":"ea82982eb46752050fa93e3096909f09fee6d29c16aaba81655357b1c385441c"}
Jul 16 21:55:12.402589 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.402567    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278" event={"ID":"b91169da-c866-4383-b08f-2dc5b7d52914","Type":"ContainerStarted","Data":"eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3"}
Jul 16 21:55:12.404593 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.404570    2555 generic.go:356] "Generic (PLEG): container finished" podID="e932528d-0c36-4366-828e-7b9a029bdfab" containerID="027a03dd616ebb79c419cd07703d478d1cead63907797ba1c87d542f0aa608df" exitCode=0
Jul 16 21:55:12.404687 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:12.404607    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm" event={"ID":"e932528d-0c36-4366-828e-7b9a029bdfab","Type":"ContainerDied","Data":"027a03dd616ebb79c419cd07703d478d1cead63907797ba1c87d542f0aa608df"}
Jul 16 21:55:13.411804 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.411772    2555 generic.go:356] "Generic (PLEG): container finished" podID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerID="9dc1dd34a27e4578cd13a1b4344d45162f0d4f1421d81a89b648e95a2fb0498f" exitCode=0
Jul 16 21:55:13.411804 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.411796    2555 generic.go:356] "Generic (PLEG): container finished" podID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerID="fc8882c41561c6fb8049f20edbda822b5e772cd4cc5fd4f3bc854d5f5b337e76" exitCode=0
Jul 16 21:55:13.412336 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.411852    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerDied","Data":"9dc1dd34a27e4578cd13a1b4344d45162f0d4f1421d81a89b648e95a2fb0498f"}
Jul 16 21:55:13.412336 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.411897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerDied","Data":"fc8882c41561c6fb8049f20edbda822b5e772cd4cc5fd4f3bc854d5f5b337e76"}
Jul 16 21:55:13.413692 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.413649    2555 generic.go:356] "Generic (PLEG): container finished" podID="b91169da-c866-4383-b08f-2dc5b7d52914" containerID="f8f76e6785c47b0f0a44d3150f7fb6771216ded4b4d69a2c313d44c618e7f917" exitCode=0
Jul 16 21:55:13.413768 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.413725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278" event={"ID":"b91169da-c866-4383-b08f-2dc5b7d52914","Type":"ContainerDied","Data":"f8f76e6785c47b0f0a44d3150f7fb6771216ded4b4d69a2c313d44c618e7f917"}
Jul 16 21:55:13.416018 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.415993    2555 generic.go:356] "Generic (PLEG): container finished" podID="e932528d-0c36-4366-828e-7b9a029bdfab" containerID="668d9f44a6889e14d3979aa10b76dd635fb3cead7c1560888f9cc29e80943549" exitCode=0
Jul 16 21:55:13.416115 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.416075    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm" event={"ID":"e932528d-0c36-4366-828e-7b9a029bdfab","Type":"ContainerDied","Data":"668d9f44a6889e14d3979aa10b76dd635fb3cead7c1560888f9cc29e80943549"}
Jul 16 21:55:13.417972 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.417950    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerID="4723cd65e22fcfbfb5759b2e677746ce29a7a315fd49429da65b15cac53d7809" exitCode=0
Jul 16 21:55:13.418049 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:13.418002    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7" event={"ID":"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13","Type":"ContainerDied","Data":"4723cd65e22fcfbfb5759b2e677746ce29a7a315fd49429da65b15cac53d7809"}
Jul 16 21:55:14.431326 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.431287    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerID="99c378e4a6949055ea9a1a2aabc97bd435ef16e9e92e01c2c9821ad559edf333" exitCode=0
Jul 16 21:55:14.431753 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.431372    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7" event={"ID":"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13","Type":"ContainerDied","Data":"99c378e4a6949055ea9a1a2aabc97bd435ef16e9e92e01c2c9821ad559edf333"}
Jul 16 21:55:14.433443 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.433420    2555 generic.go:356] "Generic (PLEG): container finished" podID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerID="9638580a5be68e0e418fee467042ff2069b2c4b9ea504634b2d9754f01bf83df" exitCode=0
Jul 16 21:55:14.433570 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.433503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerDied","Data":"9638580a5be68e0e418fee467042ff2069b2c4b9ea504634b2d9754f01bf83df"}
Jul 16 21:55:14.435335 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.435317    2555 generic.go:356] "Generic (PLEG): container finished" podID="b91169da-c866-4383-b08f-2dc5b7d52914" containerID="abc894c75c0c9357c0b5d759823eabe9ecbd0407758368dfe0a546fc0cc45983" exitCode=0
Jul 16 21:55:14.435422 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.435376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278" event={"ID":"b91169da-c866-4383-b08f-2dc5b7d52914","Type":"ContainerDied","Data":"abc894c75c0c9357c0b5d759823eabe9ecbd0407758368dfe0a546fc0cc45983"}
Jul 16 21:55:14.567198 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.567177    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:14.719828 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.719745    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-77b6v\" (UniqueName: \"kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v\") pod \"e932528d-0c36-4366-828e-7b9a029bdfab\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") "
Jul 16 21:55:14.719828 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.719810    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle\") pod \"e932528d-0c36-4366-828e-7b9a029bdfab\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") "
Jul 16 21:55:14.720026 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.719878    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util\") pod \"e932528d-0c36-4366-828e-7b9a029bdfab\" (UID: \"e932528d-0c36-4366-828e-7b9a029bdfab\") "
Jul 16 21:55:14.720306 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.720266    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle" (OuterVolumeSpecName: "bundle") pod "e932528d-0c36-4366-828e-7b9a029bdfab" (UID: "e932528d-0c36-4366-828e-7b9a029bdfab"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:14.722222 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.722199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v" (OuterVolumeSpecName: "kube-api-access-77b6v") pod "e932528d-0c36-4366-828e-7b9a029bdfab" (UID: "e932528d-0c36-4366-828e-7b9a029bdfab"). InnerVolumeSpecName "kube-api-access-77b6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:14.725922 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.725901    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util" (OuterVolumeSpecName: "util") pod "e932528d-0c36-4366-828e-7b9a029bdfab" (UID: "e932528d-0c36-4366-828e-7b9a029bdfab"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:14.821368 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.821334    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:14.821368 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.821359    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/e932528d-0c36-4366-828e-7b9a029bdfab-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:14.821368 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:14.821368    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77b6v\" (UniqueName: \"kubernetes.io/projected/e932528d-0c36-4366-828e-7b9a029bdfab-kube-api-access-77b6v\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.442401 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.442374    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm"
Jul 16 21:55:15.442401 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.442364    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm" event={"ID":"e932528d-0c36-4366-828e-7b9a029bdfab","Type":"ContainerDied","Data":"a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d"}
Jul 16 21:55:15.442886 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.442427    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a870c92e946c449191cf99d01798270ffe323025154e86271fee36484dfe9f2d"
Jul 16 21:55:15.586577 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.586553    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:55:15.624269 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.624246    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:15.628135 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.628117    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:15.728222 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728194    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle\") pod \"b91169da-c866-4383-b08f-2dc5b7d52914\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") "
Jul 16 21:55:15.728371 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util\") pod \"02182b5c-0f36-4b77-945c-cc1c8c36305c\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") "
Jul 16 21:55:15.728371 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728289    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xz2gf\" (UniqueName: \"kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf\") pod \"b91169da-c866-4383-b08f-2dc5b7d52914\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") "
Jul 16 21:55:15.728371 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728324    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle\") pod \"02182b5c-0f36-4b77-945c-cc1c8c36305c\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") "
Jul 16 21:55:15.728371 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728359    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle\") pod \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") "
Jul 16 21:55:15.728648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728385    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tt98m\" (UniqueName: \"kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m\") pod \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") "
Jul 16 21:55:15.728648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728427    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util\") pod \"b91169da-c866-4383-b08f-2dc5b7d52914\" (UID: \"b91169da-c866-4383-b08f-2dc5b7d52914\") "
Jul 16 21:55:15.728648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728530    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgjnh\" (UniqueName: \"kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh\") pod \"02182b5c-0f36-4b77-945c-cc1c8c36305c\" (UID: \"02182b5c-0f36-4b77-945c-cc1c8c36305c\") "
Jul 16 21:55:15.728648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.728579    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util\") pod \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\" (UID: \"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13\") "
Jul 16 21:55:15.729574 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.729546    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle" (OuterVolumeSpecName: "bundle") pod "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" (UID: "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.729741 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.729703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle" (OuterVolumeSpecName: "bundle") pod "02182b5c-0f36-4b77-945c-cc1c8c36305c" (UID: "02182b5c-0f36-4b77-945c-cc1c8c36305c"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.730062 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.729696    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle" (OuterVolumeSpecName: "bundle") pod "b91169da-c866-4383-b08f-2dc5b7d52914" (UID: "b91169da-c866-4383-b08f-2dc5b7d52914"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.731433 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.731393    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf" (OuterVolumeSpecName: "kube-api-access-xz2gf") pod "b91169da-c866-4383-b08f-2dc5b7d52914" (UID: "b91169da-c866-4383-b08f-2dc5b7d52914"). InnerVolumeSpecName "kube-api-access-xz2gf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:15.731570 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.731432    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m" (OuterVolumeSpecName: "kube-api-access-tt98m") pod "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" (UID: "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13"). InnerVolumeSpecName "kube-api-access-tt98m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:15.732048 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.732027    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh" (OuterVolumeSpecName: "kube-api-access-wgjnh") pod "02182b5c-0f36-4b77-945c-cc1c8c36305c" (UID: "02182b5c-0f36-4b77-945c-cc1c8c36305c"). InnerVolumeSpecName "kube-api-access-wgjnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:15.735314 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.735291    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util" (OuterVolumeSpecName: "util") pod "02182b5c-0f36-4b77-945c-cc1c8c36305c" (UID: "02182b5c-0f36-4b77-945c-cc1c8c36305c"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.735687 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.735663    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util" (OuterVolumeSpecName: "util") pod "b91169da-c866-4383-b08f-2dc5b7d52914" (UID: "b91169da-c866-4383-b08f-2dc5b7d52914"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.736443 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.736381    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util" (OuterVolumeSpecName: "util") pod "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" (UID: "6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:15.829873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829830    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.829873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829868    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.829873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829877    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xz2gf\" (UniqueName: \"kubernetes.io/projected/b91169da-c866-4383-b08f-2dc5b7d52914-kube-api-access-xz2gf\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.829873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829886    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/02182b5c-0f36-4b77-945c-cc1c8c36305c-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.829873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829895    2555 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.830216 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829902    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tt98m\" (UniqueName: \"kubernetes.io/projected/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-kube-api-access-tt98m\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.830216 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829910    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/b91169da-c866-4383-b08f-2dc5b7d52914-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.830216 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829918    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgjnh\" (UniqueName: \"kubernetes.io/projected/02182b5c-0f36-4b77-945c-cc1c8c36305c-kube-api-access-wgjnh\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:15.830216 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:15.829925    2555 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13-util\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:55:16.448873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.448842    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7"
Jul 16 21:55:16.448873 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.448859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7" event={"ID":"6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13","Type":"ContainerDied","Data":"d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f"}
Jul 16 21:55:16.449432 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.448904    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d0e82e84fb1c60d70ff19688a6ce37397b71ce12199c3c4441851111e888720f"
Jul 16 21:55:16.450741 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.450725    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk"
Jul 16 21:55:16.450854 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.450726    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk" event={"ID":"02182b5c-0f36-4b77-945c-cc1c8c36305c","Type":"ContainerDied","Data":"98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02"}
Jul 16 21:55:16.450854 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.450837    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98f630c1ffa045fda60b586977109bdb442d5c088ba1ad38e032e90e5dd9ad02"
Jul 16 21:55:16.452720 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.452703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278" event={"ID":"b91169da-c866-4383-b08f-2dc5b7d52914","Type":"ContainerDied","Data":"eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3"}
Jul 16 21:55:16.452813 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.452730    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee2b37b4f37c8e4d77370589d01f982221d1f3b2807a738210dc10af924a9b3"
Jul 16 21:55:16.452813 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:55:16.452731    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278"
Jul 16 21:56:05.279619 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:05.279587    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:56:28.046543 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046453    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046863    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046884    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046896    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046902    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046921    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046930    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046939    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046946    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="extract"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046961    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="pull"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046969    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="pull"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046976    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046983    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046989    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.046994    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047000    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047007    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047016    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="util"
Jul 16 21:56:28.047011 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047021    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="util"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047030    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047036    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047042    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047046    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047053    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047057    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="pull"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047116    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e932528d-0c36-4366-828e-7b9a029bdfab" containerName="extract"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047123    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="02182b5c-0f36-4b77-945c-cc1c8c36305c" containerName="extract"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047130    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13" containerName="extract"
Jul 16 21:56:28.047595 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.047136    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b91169da-c866-4383-b08f-2dc5b7d52914" containerName="extract"
Jul 16 21:56:28.050235 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.050221    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.052805 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.052784    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 16 21:56:28.053651 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.053627    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-cbks6\""
Jul 16 21:56:28.053651 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.053646    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 16 21:56:28.053825 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.053661    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 16 21:56:28.061254 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.061230    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:56:28.087172 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.087143    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:56:28.167862 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.167827    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71e0fe90-a227-40bf-881d-60a8a40bf654-config-file\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.168030 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.167922    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/71e0fe90-a227-40bf-881d-60a8a40bf654-kube-api-access-nrkbf\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.268594 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.268561    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/71e0fe90-a227-40bf-881d-60a8a40bf654-kube-api-access-nrkbf\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.268758 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.268608    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71e0fe90-a227-40bf-881d-60a8a40bf654-config-file\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.269191 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.269172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/71e0fe90-a227-40bf-881d-60a8a40bf654-config-file\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.281026 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.280997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrkbf\" (UniqueName: \"kubernetes.io/projected/71e0fe90-a227-40bf-881d-60a8a40bf654-kube-api-access-nrkbf\") pod \"limitador-limitador-54d6d988d5-xl5tk\" (UID: \"71e0fe90-a227-40bf-881d-60a8a40bf654\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.362217 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.362128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:28.502397 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.502370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:56:28.504357 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:56:28.504330    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e0fe90_a227_40bf_881d_60a8a40bf654.slice/crio-ef044a0b1dada8801119b608a194775aeffcc2d4de8f27ed7a3670cd9ed4ef2c WatchSource:0}: Error finding container ef044a0b1dada8801119b608a194775aeffcc2d4de8f27ed7a3670cd9ed4ef2c: Status 404 returned error can't find the container with id ef044a0b1dada8801119b608a194775aeffcc2d4de8f27ed7a3670cd9ed4ef2c
Jul 16 21:56:28.822779 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:28.822742    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk" event={"ID":"71e0fe90-a227-40bf-881d-60a8a40bf654","Type":"ContainerStarted","Data":"ef044a0b1dada8801119b608a194775aeffcc2d4de8f27ed7a3670cd9ed4ef2c"}
Jul 16 21:56:30.299785 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:30.299740    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-58d7d65fc4-zpkwt" podUID="53bfcfee-81ee-460c-9b83-3602082799c0" containerName="console" containerID="cri-o://b089e0b113228740cd82a6b27f10c7d348bee60b89494caf44809f61df695f23" gracePeriod=15
Jul 16 21:56:30.833715 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:30.833636    2555 generic.go:356] "Generic (PLEG): container finished" podID="53bfcfee-81ee-460c-9b83-3602082799c0" containerID="b089e0b113228740cd82a6b27f10c7d348bee60b89494caf44809f61df695f23" exitCode=0
Jul 16 21:56:30.833847 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:30.833705    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d7d65fc4-zpkwt" event={"ID":"53bfcfee-81ee-460c-9b83-3602082799c0","Type":"ContainerDied","Data":"b089e0b113228740cd82a6b27f10c7d348bee60b89494caf44809f61df695f23"}
Jul 16 21:56:31.047589 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.047569    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:56:31.093968 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.093949    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094086 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.093982    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094086 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094002    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094086 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094029    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094086 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094064    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094275 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094159    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094275 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094253    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whhbm\" (UniqueName: \"kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm\") pod \"53bfcfee-81ee-460c-9b83-3602082799c0\" (UID: \"53bfcfee-81ee-460c-9b83-3602082799c0\") "
Jul 16 21:56:31.094404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094377    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:31.094482 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094436    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca" (OuterVolumeSpecName: "service-ca") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:31.094571 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094554    2555 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-trusted-ca-bundle\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.094642 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094577    2555 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.094955 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094896    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config" (OuterVolumeSpecName: "console-config") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:31.094955 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.094951    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:56:31.096642 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.096609    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:56:31.097138 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.097113    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:56:31.097240 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.097146    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm" (OuterVolumeSpecName: "kube-api-access-whhbm") pod "53bfcfee-81ee-460c-9b83-3602082799c0" (UID: "53bfcfee-81ee-460c-9b83-3602082799c0"). InnerVolumeSpecName "kube-api-access-whhbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:56:31.195420 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.195390    2555 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-oauth-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.195420 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.195415    2555 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-oauth-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.195420 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.195424    2555 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/53bfcfee-81ee-460c-9b83-3602082799c0-console-serving-cert\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.195674 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.195433    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whhbm\" (UniqueName: \"kubernetes.io/projected/53bfcfee-81ee-460c-9b83-3602082799c0-kube-api-access-whhbm\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.195674 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.195442    2555 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/53bfcfee-81ee-460c-9b83-3602082799c0-console-config\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 21:56:31.840617 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.840575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk" event={"ID":"71e0fe90-a227-40bf-881d-60a8a40bf654","Type":"ContainerStarted","Data":"66cb81d5b210b15d44b6fdbef77db167c1d7dc2471f25692fd6c33b11fb868bf"}
Jul 16 21:56:31.841027 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.840716    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:56:31.841954 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.841931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-58d7d65fc4-zpkwt" event={"ID":"53bfcfee-81ee-460c-9b83-3602082799c0","Type":"ContainerDied","Data":"f1302b60251d283b41f16663d2a4d7584ce934c73cd5fca84ad261ca3aace728"}
Jul 16 21:56:31.842075 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.841979    2555 scope.go:117] "RemoveContainer" containerID="b089e0b113228740cd82a6b27f10c7d348bee60b89494caf44809f61df695f23"
Jul 16 21:56:31.842075 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.841937    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-58d7d65fc4-zpkwt"
Jul 16 21:56:31.861284 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.861241    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk" podStartSLOduration=1.283502305 podStartE2EDuration="3.861225217s" totalImagesPullingTime="2.577722912s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:31.859530461 +0000 UTC m=+446.696584261" watchObservedRunningTime="2026-07-16 21:56:31.861225217 +0000 UTC m=+446.698279017"
Jul 16 21:56:31.880263 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.880236    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:56:31.887494 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:31.887447    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-58d7d65fc4-zpkwt"]
Jul 16 21:56:33.678178 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:33.678146    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53bfcfee-81ee-460c-9b83-3602082799c0" path="/var/lib/kubelet/pods/53bfcfee-81ee-460c-9b83-3602082799c0/volumes"
Jul 16 21:56:42.849351 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:56:42.849324    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"
Jul 16 21:57:46.207096 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:57:46.207057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:58:08.386851 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:08.386777    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:58:18.287678 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:18.287641    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:58:24.574670 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.574637    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"]
Jul 16 21:58:24.575293 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.575273    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53bfcfee-81ee-460c-9b83-3602082799c0" containerName="console"
Jul 16 21:58:24.575293 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.575295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="53bfcfee-81ee-460c-9b83-3602082799c0" containerName="console"
Jul 16 21:58:24.575500 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.575388    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="53bfcfee-81ee-460c-9b83-3602082799c0" containerName="console"
Jul 16 21:58:24.577781 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.577762    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.580343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.580326    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 16 21:58:24.581369 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.581348    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Jul 16 21:58:24.581430 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.581353    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 16 21:58:24.581495 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.581352    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-khxnq\""
Jul 16 21:58:24.590584 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.590561    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"]
Jul 16 21:58:24.668679 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668649    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.668679 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668691    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.668906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668734    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.668906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9946713d-97f7-42b8-b56b-92283a1ecc0e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.668906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668868    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.668906 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.668894    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6zs8n\" (UniqueName: \"kubernetes.io/projected/9946713d-97f7-42b8-b56b-92283a1ecc0e-kube-api-access-6zs8n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770071 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770028    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770071 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770074    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770104    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770128    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9946713d-97f7-42b8-b56b-92283a1ecc0e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770151    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770343 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770168    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6zs8n\" (UniqueName: \"kubernetes.io/projected/9946713d-97f7-42b8-b56b-92283a1ecc0e-kube-api-access-6zs8n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770565 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770535    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770648 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770618    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.770777 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.770753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.772541 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.772519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9946713d-97f7-42b8-b56b-92283a1ecc0e-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.772677 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.772663    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9946713d-97f7-42b8-b56b-92283a1ecc0e-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.779127 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.779104    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6zs8n\" (UniqueName: \"kubernetes.io/projected/9946713d-97f7-42b8-b56b-92283a1ecc0e-kube-api-access-6zs8n\") pod \"e2e-trlp-test-simulated-kserve-598466548d-dsrk2\" (UID: \"9946713d-97f7-42b8-b56b-92283a1ecc0e\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:24.888785 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:24.888703    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:25.026949 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:25.026912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"]
Jul 16 21:58:25.028518 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:58:25.028488    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9946713d_97f7_42b8_b56b_92283a1ecc0e.slice/crio-94a3608f23b276731d99016ae6c6d3dfb4f5ad740f8d150b7f615e0d069dff2a WatchSource:0}: Error finding container 94a3608f23b276731d99016ae6c6d3dfb4f5ad740f8d150b7f615e0d069dff2a: Status 404 returned error can't find the container with id 94a3608f23b276731d99016ae6c6d3dfb4f5ad740f8d150b7f615e0d069dff2a
Jul 16 21:58:25.406614 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:25.406572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2" event={"ID":"9946713d-97f7-42b8-b56b-92283a1ecc0e","Type":"ContainerStarted","Data":"94a3608f23b276731d99016ae6c6d3dfb4f5ad740f8d150b7f615e0d069dff2a"}
Jul 16 21:58:25.783647 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:25.783616    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:58:26.418560 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:26.418455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2" event={"ID":"9946713d-97f7-42b8-b56b-92283a1ecc0e","Type":"ContainerStarted","Data":"1f0c5113cf67f594073ce7a0865e7bafe75f823878be3e9efac6c85172b877f6"}
Jul 16 21:58:26.437819 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:26.437763    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2" podStartSLOduration=1.461903454 podStartE2EDuration="2.437748557s" totalImagesPullingTime="975.845103ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:26.435438976 +0000 UTC m=+561.272492789" watchObservedRunningTime="2026-07-16 21:58:26.437748557 +0000 UTC m=+561.274802379"
Jul 16 21:58:34.889645 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:34.889611    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:34.890039 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:34.889696    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:34.902404 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:34.902375    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:35.474141 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:35.474107    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-dsrk2"
Jul 16 21:58:36.488953 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.488920    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"]
Jul 16 21:58:36.491711 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.491689    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.493795 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.493775    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Jul 16 21:58:36.502928 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.502904    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"]
Jul 16 21:58:36.575226 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.575411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575233    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8gs6\" (UniqueName: \"kubernetes.io/projected/252c0036-d381-4c71-8450-e1b066cef4a4-kube-api-access-b8gs6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.575411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575289    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.575411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252c0036-d381-4c71-8450-e1b066cef4a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.575411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575370    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.575411 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.575393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.676696 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.676662    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252c0036-d381-4c71-8450-e1b066cef4a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.676864 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.676717    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.676864 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.676844    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.676949 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.676907    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.676997 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.676942    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8gs6\" (UniqueName: \"kubernetes.io/projected/252c0036-d381-4c71-8450-e1b066cef4a4-kube-api-access-b8gs6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.677060 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.677010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.677115 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.677070    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.677291 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.677263    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.677291 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.677287    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.679137 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.679114    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/252c0036-d381-4c71-8450-e1b066cef4a4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.679442 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.679426    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/252c0036-d381-4c71-8450-e1b066cef4a4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.684793 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.684775    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8gs6\" (UniqueName: \"kubernetes.io/projected/252c0036-d381-4c71-8450-e1b066cef4a4-kube-api-access-b8gs6\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk\" (UID: \"252c0036-d381-4c71-8450-e1b066cef4a4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.802125 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.802093    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:36.939067 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:36.939046    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"]
Jul 16 21:58:36.941285 ip-10-0-131-79 kubenswrapper[2555]: W0716 21:58:36.941259    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252c0036_d381_4c71_8450_e1b066cef4a4.slice/crio-fcf7852ce2f7d15f010ad150e4d61e510fb45281e9dd6dd3c406be72a4f36562 WatchSource:0}: Error finding container fcf7852ce2f7d15f010ad150e4d61e510fb45281e9dd6dd3c406be72a4f36562: Status 404 returned error can't find the container with id fcf7852ce2f7d15f010ad150e4d61e510fb45281e9dd6dd3c406be72a4f36562
Jul 16 21:58:37.475947 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:37.475859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk" event={"ID":"252c0036-d381-4c71-8450-e1b066cef4a4","Type":"ContainerStarted","Data":"81cacc68670b0ccd98bc77e7e8b8d32e1cdb66914dfcc6a7fe6b6e8dfb063db7"}
Jul 16 21:58:37.475947 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:37.475920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk" event={"ID":"252c0036-d381-4c71-8450-e1b066cef4a4","Type":"ContainerStarted","Data":"fcf7852ce2f7d15f010ad150e4d61e510fb45281e9dd6dd3c406be72a4f36562"}
Jul 16 21:58:37.502333 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:37.502282    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk" podStartSLOduration=1.319608517 podStartE2EDuration="1.502267087s" totalImagesPullingTime="182.65857ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:37.498687243 +0000 UTC m=+572.335741053" watchObservedRunningTime="2026-07-16 21:58:37.502267087 +0000 UTC m=+572.339320888"
Jul 16 21:58:38.077376 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:38.077338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:58:46.802743 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:46.802708    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:46.803166 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:46.802781    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:46.815292 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:46.815268    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:47.538329 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:47.538300    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk"
Jul 16 21:58:55.679961 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:58:55.679929    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 21:59:10.682710 ip-10-0-131-79 kubenswrapper[2555]: I0716 21:59:10.682674    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:00:40.591509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:00:40.591456    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:00:50.191682 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:00:50.191648    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:01:00.894336 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:01:00.894243    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:02:43.592592 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:02:43.592557    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:02:53.366140 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:02:53.366101    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:03:27.217416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:03:27.217378    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:03:36.285414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:03:36.285375    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:03:45.805497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:03:45.804737    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:03:54.214785 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:03:54.214751    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:04:04.090872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:04:04.090795    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:04:19.912862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:04:19.912825    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:04:27.384396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:04:27.384351    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:04:34.988398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:04:34.988361    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:04:59.797794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:04:59.797750    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:05:11.494941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:05:11.494903    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:05:34.796314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:05:34.796230    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:05:45.592524 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:05:45.592490    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:06:00.692449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:06:00.692408    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:06:10.294905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:06:10.294868    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:06:19.990499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:06:19.990444    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:06:29.089025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:06:29.088987    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:06:40.590204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:06:40.590166    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:08:28.991332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:08:28.991248    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:08:40.786971 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:08:40.786936    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:08:49.586841 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:08:49.586810    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:04.189324 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:04.189282    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:14.280693 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:14.280655    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:23.692985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:23.692942    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:32.391171 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:32.391134    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:43.603630 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:43.603597    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:09:55.284865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:09:55.284827    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:10:04.311385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:10:04.311347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:10:17.688291 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:10:17.688254    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:10:38.481828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:10:38.481786    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:10:48.000820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:10:48.000787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:10:57.994188 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:10:57.994149    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:11:10.292557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:11:10.292523    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:11:28.309251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:11:28.309175    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:11:37.694715 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:11:37.694679    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:11:47.092030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:11:47.091992    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:11:57.484904 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:11:57.484860    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:12:06.982397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:12:06.982358    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:12:16.588805 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:12:16.588761    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:12:27.194796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:12:27.194757    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:12:39.178673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:12:39.178628    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:12:49.099526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:12:49.099490    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:01.695510 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:01.695419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:11.677895 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:11.677859    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:18.489479 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:18.489425    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:34.687326 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:34.687289    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:44.805616 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:44.805577    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:13:54.398379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:13:54.398344    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:03.291243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:03.291208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:12.910454 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:12.910419    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:25.489514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:25.489476    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:36.995719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:36.995684    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:49.249083 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.249049    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:14:49.252391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.252374    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:14:49.266120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.266098    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:14:49.266614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.266600    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:14:49.292502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.292458    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:14:49.361109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.361071    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf\" (UID: \"b786c12d-8185-46be-8778-6a246aa8b904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:14:49.462029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.461990    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf\" (UID: \"b786c12d-8185-46be-8778-6a246aa8b904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:14:49.462640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.462621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf\" (UID: \"b786c12d-8185-46be-8778-6a246aa8b904\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:14:49.561992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.561948    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:14:49.713610 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.713581    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:14:49.714181 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:14:49.714156    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb786c12d_8185_46be_8778_6a246aa8b904.slice/crio-2fc46dd5e116f2a3c50ca908504c200d29cb012075b4dcd77b25bce05e1914ce WatchSource:0}: Error finding container 2fc46dd5e116f2a3c50ca908504c200d29cb012075b4dcd77b25bce05e1914ce: Status 404 returned error can't find the container with id 2fc46dd5e116f2a3c50ca908504c200d29cb012075b4dcd77b25bce05e1914ce
Jul 16 22:14:49.716133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:49.716117    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:14:50.411861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:50.411829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerStarted","Data":"2fc46dd5e116f2a3c50ca908504c200d29cb012075b4dcd77b25bce05e1914ce"}
Jul 16 22:14:51.006497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:51.006452    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:14:52.423906 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:52.423868    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerStarted","Data":"22dbebc1ccf1a2f3a96839448d73e8c75206282172c21387971430238fddc72f"}
Jul 16 22:14:52.455090 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:52.455038    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" podStartSLOduration=1.168671654 podStartE2EDuration="3.455021495s" totalImagesPullingTime="2.286349841s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:14:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:14:52.452534686 +0000 UTC m=+1547.289588489" watchObservedRunningTime="2026-07-16 22:14:52.455021495 +0000 UTC m=+1547.292075296"
Jul 16 22:14:53.430197 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:53.430167    2555 generic.go:356] "Generic (PLEG): container finished" podID="b786c12d-8185-46be-8778-6a246aa8b904" containerID="22dbebc1ccf1a2f3a96839448d73e8c75206282172c21387971430238fddc72f" exitCode=7
Jul 16 22:14:53.430621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:53.430256    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerDied","Data":"22dbebc1ccf1a2f3a96839448d73e8c75206282172c21387971430238fddc72f"}
Jul 16 22:14:53.430621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:53.430418    2555 scope.go:117] "RemoveContainer" containerID="22dbebc1ccf1a2f3a96839448d73e8c75206282172c21387971430238fddc72f"
Jul 16 22:14:54.437024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:54.436988    2555 generic.go:356] "Generic (PLEG): container finished" podID="b786c12d-8185-46be-8778-6a246aa8b904" containerID="ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b" exitCode=7
Jul 16 22:14:54.437649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:54.437071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerDied","Data":"ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b"}
Jul 16 22:14:54.437649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:54.437134    2555 scope.go:117] "RemoveContainer" containerID="22dbebc1ccf1a2f3a96839448d73e8c75206282172c21387971430238fddc72f"
Jul 16 22:14:54.437649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:54.437290    2555 scope.go:117] "RemoveContainer" containerID="ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b"
Jul 16 22:14:54.437649 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:14:54.437547    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf_odh-ai-gateway-infra(b786c12d-8185-46be-8778-6a246aa8b904)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" podUID="b786c12d-8185-46be-8778-6a246aa8b904"
Jul 16 22:14:55.449669 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:14:55.444634    2555 scope.go:117] "RemoveContainer" containerID="ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b"
Jul 16 22:14:55.449669 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:14:55.445057    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf_odh-ai-gateway-infra(b786c12d-8185-46be-8778-6a246aa8b904)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" podUID="b786c12d-8185-46be-8778-6a246aa8b904"
Jul 16 22:15:10.670816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:10.670788    2555 scope.go:117] "RemoveContainer" containerID="ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b"
Jul 16 22:15:11.527485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:11.527418    2555 generic.go:356] "Generic (PLEG): container finished" podID="b786c12d-8185-46be-8778-6a246aa8b904" containerID="38c3e777fa3357209ecccfee4f572771bf762976b38370fd8251af4619730cab" exitCode=7
Jul 16 22:15:11.527663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:11.527491    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerDied","Data":"38c3e777fa3357209ecccfee4f572771bf762976b38370fd8251af4619730cab"}
Jul 16 22:15:11.527663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:11.527585    2555 scope.go:117] "RemoveContainer" containerID="ea62a3e53f0ebd0aa362ed7c6e82c65ecb2a3edd4031729931a29944414f189b"
Jul 16 22:15:11.527940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:11.527915    2555 scope.go:117] "RemoveContainer" containerID="38c3e777fa3357209ecccfee4f572771bf762976b38370fd8251af4619730cab"
Jul 16 22:15:11.528198 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:15:11.528170    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf_odh-ai-gateway-infra(b786c12d-8185-46be-8778-6a246aa8b904)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" podUID="b786c12d-8185-46be-8778-6a246aa8b904"
Jul 16 22:15:12.555345 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:12.555311    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:15:12.690121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:12.690098    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:15:12.758234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:12.758201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca\") pod \"b786c12d-8185-46be-8778-6a246aa8b904\" (UID: \"b786c12d-8185-46be-8778-6a246aa8b904\") "
Jul 16 22:15:12.760557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:12.758842    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b786c12d-8185-46be-8778-6a246aa8b904" (UID: "b786c12d-8185-46be-8778-6a246aa8b904"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:15:12.760557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:12.759096    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b786c12d-8185-46be-8778-6a246aa8b904-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:15:13.544082 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.544051    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"
Jul 16 22:15:13.544082 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.544055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf" event={"ID":"b786c12d-8185-46be-8778-6a246aa8b904","Type":"ContainerDied","Data":"2fc46dd5e116f2a3c50ca908504c200d29cb012075b4dcd77b25bce05e1914ce"}
Jul 16 22:15:13.544344 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.544157    2555 scope.go:117] "RemoveContainer" containerID="38c3e777fa3357209ecccfee4f572771bf762976b38370fd8251af4619730cab"
Jul 16 22:15:13.570434 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.570405    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:15:13.580005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.579982    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-p6qcf"]
Jul 16 22:15:13.675704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:13.675678    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b786c12d-8185-46be-8778-6a246aa8b904" path="/var/lib/kubelet/pods/b786c12d-8185-46be-8778-6a246aa8b904/volumes"
Jul 16 22:15:49.268478 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268426    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268773    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268785    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268807    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268813    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268863    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.268950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.268871    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:15:49.271882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.271867    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:15:49.274729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.274700    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:15:49.274729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.274722    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:15:49.280085 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.280065    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:15:49.387576 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.387543    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8\" (UID: \"761796a1-f940-4ca3-b4f5-6a2b15cba210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:15:49.488695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.488653    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8\" (UID: \"761796a1-f940-4ca3-b4f5-6a2b15cba210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:15:49.489319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.489298    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8\" (UID: \"761796a1-f940-4ca3-b4f5-6a2b15cba210\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:15:49.583881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.583847    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:15:49.719253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.719225    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:15:49.721214 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:15:49.721185    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod761796a1_f940_4ca3_b4f5_6a2b15cba210.slice/crio-666f95993b7406fa6a8670643b2e32c13bdcefb5de61fbb799a150141f476f76 WatchSource:0}: Error finding container 666f95993b7406fa6a8670643b2e32c13bdcefb5de61fbb799a150141f476f76: Status 404 returned error can't find the container with id 666f95993b7406fa6a8670643b2e32c13bdcefb5de61fbb799a150141f476f76
Jul 16 22:15:49.729733 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:49.729707    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" event={"ID":"761796a1-f940-4ca3-b4f5-6a2b15cba210","Type":"ContainerStarted","Data":"666f95993b7406fa6a8670643b2e32c13bdcefb5de61fbb799a150141f476f76"}
Jul 16 22:15:50.739970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:50.739878    2555 generic.go:356] "Generic (PLEG): container finished" podID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerID="9355102bb377740666d5edad4f4dc9a1c17aa6ba23f63533d0883e0b028297ad" exitCode=7
Jul 16 22:15:50.739970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:50.739957    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" event={"ID":"761796a1-f940-4ca3-b4f5-6a2b15cba210","Type":"ContainerDied","Data":"9355102bb377740666d5edad4f4dc9a1c17aa6ba23f63533d0883e0b028297ad"}
Jul 16 22:15:50.740385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:50.740173    2555 scope.go:117] "RemoveContainer" containerID="9355102bb377740666d5edad4f4dc9a1c17aa6ba23f63533d0883e0b028297ad"
Jul 16 22:15:51.746587 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:51.746498    2555 generic.go:356] "Generic (PLEG): container finished" podID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerID="0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271" exitCode=7
Jul 16 22:15:51.747012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:51.746585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" event={"ID":"761796a1-f940-4ca3-b4f5-6a2b15cba210","Type":"ContainerDied","Data":"0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271"}
Jul 16 22:15:51.747012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:51.746646    2555 scope.go:117] "RemoveContainer" containerID="9355102bb377740666d5edad4f4dc9a1c17aa6ba23f63533d0883e0b028297ad"
Jul 16 22:15:51.747012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:51.746754    2555 scope.go:117] "RemoveContainer" containerID="0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271"
Jul 16 22:15:51.747012 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:15:51.746964    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8_odh-ai-gateway-infra(761796a1-f940-4ca3-b4f5-6a2b15cba210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210"
Jul 16 22:15:52.753868 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:15:52.753841    2555 scope.go:117] "RemoveContainer" containerID="0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271"
Jul 16 22:15:52.754262 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:15:52.754035    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8_odh-ai-gateway-infra(761796a1-f940-4ca3-b4f5-6a2b15cba210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210"
Jul 16 22:16:03.670938 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:03.670855    2555 scope.go:117] "RemoveContainer" containerID="0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271"
Jul 16 22:16:04.814227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:04.814199    2555 generic.go:356] "Generic (PLEG): container finished" podID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerID="b98d275580278d124db03cf082b0b165e00ea66ff601db934692ea8f1fa0f74a" exitCode=7
Jul 16 22:16:04.814705 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:04.814279    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" event={"ID":"761796a1-f940-4ca3-b4f5-6a2b15cba210","Type":"ContainerDied","Data":"b98d275580278d124db03cf082b0b165e00ea66ff601db934692ea8f1fa0f74a"}
Jul 16 22:16:04.814705 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:04.814357    2555 scope.go:117] "RemoveContainer" containerID="0646cd82f0faf6145267b2f372d0ada956ecd8eea66d0f27f7e9926a1621d271"
Jul 16 22:16:04.814820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:04.814752    2555 scope.go:117] "RemoveContainer" containerID="b98d275580278d124db03cf082b0b165e00ea66ff601db934692ea8f1fa0f74a"
Jul 16 22:16:04.814952 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:16:04.814933    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8_odh-ai-gateway-infra(761796a1-f940-4ca3-b4f5-6a2b15cba210)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210"
Jul 16 22:16:05.844648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:05.844614    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:16:05.982381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:05.982356    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:16:06.138330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.138296    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca\") pod \"761796a1-f940-4ca3-b4f5-6a2b15cba210\" (UID: \"761796a1-f940-4ca3-b4f5-6a2b15cba210\") "
Jul 16 22:16:06.138720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.138698    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "761796a1-f940-4ca3-b4f5-6a2b15cba210" (UID: "761796a1-f940-4ca3-b4f5-6a2b15cba210"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:16:06.239841 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.239807    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/761796a1-f940-4ca3-b4f5-6a2b15cba210-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:16:06.827867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.827841    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"
Jul 16 22:16:06.827867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.827857    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8" event={"ID":"761796a1-f940-4ca3-b4f5-6a2b15cba210","Type":"ContainerDied","Data":"666f95993b7406fa6a8670643b2e32c13bdcefb5de61fbb799a150141f476f76"}
Jul 16 22:16:06.828103 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.827913    2555 scope.go:117] "RemoveContainer" containerID="b98d275580278d124db03cf082b0b165e00ea66ff601db934692ea8f1fa0f74a"
Jul 16 22:16:06.850680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.850652    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:16:06.857158 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:06.857128    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4kcd8"]
Jul 16 22:16:07.676061 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:07.676028    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" path="/var/lib/kubelet/pods/761796a1-f940-4ca3-b4f5-6a2b15cba210/volumes"
Jul 16 22:16:39.306086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306048    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306500    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306518    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306544    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306552    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306567    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:16:39.306621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306577    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:16:39.306881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306664    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306677    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:16:39.306881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.306689    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b786c12d-8185-46be-8778-6a246aa8b904" containerName="revoke-keys"
Jul 16 22:16:39.309729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.309713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:16:39.312391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.312372    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:16:39.312511 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.312395    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:16:39.317664 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.317170    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:16:39.409132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.409097    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m\" (UID: \"b4409e63-00b5-4b1d-91ce-76fd177a28ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:16:39.509517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.509474    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m\" (UID: \"b4409e63-00b5-4b1d-91ce-76fd177a28ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:16:39.510018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.510001    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m\" (UID: \"b4409e63-00b5-4b1d-91ce-76fd177a28ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:16:39.621154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.621069    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:16:39.769891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:39.769860    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:16:39.771505 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:16:39.771452    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4409e63_00b5_4b1d_91ce_76fd177a28ff.slice/crio-8237e075d71ca67871193f987290bc4a109ef703462b85d3552de24e372481ee WatchSource:0}: Error finding container 8237e075d71ca67871193f987290bc4a109ef703462b85d3552de24e372481ee: Status 404 returned error can't find the container with id 8237e075d71ca67871193f987290bc4a109ef703462b85d3552de24e372481ee
Jul 16 22:16:40.008137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:40.008050    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerStarted","Data":"8237e075d71ca67871193f987290bc4a109ef703462b85d3552de24e372481ee"}
Jul 16 22:16:41.015142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:41.015102    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerID="d41cb81ae37cad5e560a00d7658b1e358d97b09b971b968b5d571edf533020c8" exitCode=7
Jul 16 22:16:41.015557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:41.015192    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerDied","Data":"d41cb81ae37cad5e560a00d7658b1e358d97b09b971b968b5d571edf533020c8"}
Jul 16 22:16:41.015557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:41.015339    2555 scope.go:117] "RemoveContainer" containerID="d41cb81ae37cad5e560a00d7658b1e358d97b09b971b968b5d571edf533020c8"
Jul 16 22:16:42.022125 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:42.022025    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerID="b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9" exitCode=7
Jul 16 22:16:42.022125 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:42.022079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerDied","Data":"b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9"}
Jul 16 22:16:42.022672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:42.022127    2555 scope.go:117] "RemoveContainer" containerID="d41cb81ae37cad5e560a00d7658b1e358d97b09b971b968b5d571edf533020c8"
Jul 16 22:16:42.022672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:42.022333    2555 scope.go:117] "RemoveContainer" containerID="b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9"
Jul 16 22:16:42.022672 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:16:42.022632    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m_odh-ai-gateway-infra(b4409e63-00b5-4b1d-91ce-76fd177a28ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff"
Jul 16 22:16:43.029822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:43.029797    2555 scope.go:117] "RemoveContainer" containerID="b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9"
Jul 16 22:16:43.030196 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:16:43.029994    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m_odh-ai-gateway-infra(b4409e63-00b5-4b1d-91ce-76fd177a28ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff"
Jul 16 22:16:57.670525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:57.670456    2555 scope.go:117] "RemoveContainer" containerID="b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9"
Jul 16 22:16:58.111114 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:58.111079    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerStarted","Data":"a1b6bb88eef483bae64aebbaf0bcf50a18b340b32413e12dc3e450fd2a141734"}
Jul 16 22:16:58.135221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:58.135174    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" podStartSLOduration=19.135159079 podStartE2EDuration="19.135159079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:16:58.133036188 +0000 UTC m=+1672.970089989" watchObservedRunningTime="2026-07-16 22:16:58.135159079 +0000 UTC m=+1672.972212880"
Jul 16 22:16:59.117315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:59.117282    2555 generic.go:356] "Generic (PLEG): container finished" podID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerID="a1b6bb88eef483bae64aebbaf0bcf50a18b340b32413e12dc3e450fd2a141734" exitCode=7
Jul 16 22:16:59.117776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:59.117331    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerDied","Data":"a1b6bb88eef483bae64aebbaf0bcf50a18b340b32413e12dc3e450fd2a141734"}
Jul 16 22:16:59.117776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:59.117382    2555 scope.go:117] "RemoveContainer" containerID="b1599d042cfa3078f7ac9c4881219d9970aa7d64d2c4f038db22495400d84ac9"
Jul 16 22:16:59.117776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:59.117630    2555 scope.go:117] "RemoveContainer" containerID="a1b6bb88eef483bae64aebbaf0bcf50a18b340b32413e12dc3e450fd2a141734"
Jul 16 22:16:59.117910 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:16:59.117855    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m_odh-ai-gateway-infra(b4409e63-00b5-4b1d-91ce-76fd177a28ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff"
Jul 16 22:16:59.152099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:16:59.152070    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:17:00.263500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:00.263456    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:17:00.391993 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:00.391888    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca\") pod \"b4409e63-00b5-4b1d-91ce-76fd177a28ff\" (UID: \"b4409e63-00b5-4b1d-91ce-76fd177a28ff\") "
Jul 16 22:17:00.392277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:00.392254    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b4409e63-00b5-4b1d-91ce-76fd177a28ff" (UID: "b4409e63-00b5-4b1d-91ce-76fd177a28ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:00.493298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:00.493260    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b4409e63-00b5-4b1d-91ce-76fd177a28ff-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:17:01.132102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.132064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m" event={"ID":"b4409e63-00b5-4b1d-91ce-76fd177a28ff","Type":"ContainerDied","Data":"8237e075d71ca67871193f987290bc4a109ef703462b85d3552de24e372481ee"}
Jul 16 22:17:01.132102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.132098    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"
Jul 16 22:17:01.132337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.132128    2555 scope.go:117] "RemoveContainer" containerID="a1b6bb88eef483bae64aebbaf0bcf50a18b340b32413e12dc3e450fd2a141734"
Jul 16 22:17:01.166113 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.166086    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:17:01.172594 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.172568    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-jsf8m"]
Jul 16 22:17:01.676852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:01.676816    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" path="/var/lib/kubelet/pods/b4409e63-00b5-4b1d-91ce-76fd177a28ff/volumes"
Jul 16 22:17:39.354232 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354197    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354548    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354560    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354568    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354573    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354584    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354590    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354656    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354664    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="761796a1-f940-4ca3-b4f5-6a2b15cba210" containerName="revoke-keys"
Jul 16 22:17:39.354761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.354673    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:17:39.357541 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.357524    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:39.360113 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.360091    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:17:39.360283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.360268    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:17:39.366923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.366877    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:39.422709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.422675    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk\" (UID: \"5696511b-079e-43eb-b50d-a2b16716e4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:39.523049 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.523016    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk\" (UID: \"5696511b-079e-43eb-b50d-a2b16716e4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:39.523640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.523621    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk\" (UID: \"5696511b-079e-43eb-b50d-a2b16716e4f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:39.670082 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.669984    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:39.807577 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:39.807553    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:39.810128 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:17:39.810098    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5696511b_079e_43eb_b50d_a2b16716e4f0.slice/crio-a74bdd16ae99b89391717cf9f817459f9b8cb750907e832cfa9b9600adc3828c WatchSource:0}: Error finding container a74bdd16ae99b89391717cf9f817459f9b8cb750907e832cfa9b9600adc3828c: Status 404 returned error can't find the container with id a74bdd16ae99b89391717cf9f817459f9b8cb750907e832cfa9b9600adc3828c
Jul 16 22:17:40.331051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:40.331011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerStarted","Data":"af60239b0824673a94f249cc5c6c78634af4d57985275e7a98b0eadf1c360662"}
Jul 16 22:17:40.331256 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:40.331070    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerStarted","Data":"a74bdd16ae99b89391717cf9f817459f9b8cb750907e832cfa9b9600adc3828c"}
Jul 16 22:17:40.349880 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:40.349826    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" podStartSLOduration=1.349811265 podStartE2EDuration="1.349811265s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:17:40.347893717 +0000 UTC m=+1715.184947529" watchObservedRunningTime="2026-07-16 22:17:40.349811265 +0000 UTC m=+1715.186865068"
Jul 16 22:17:41.336863 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:41.336826    2555 generic.go:356] "Generic (PLEG): container finished" podID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerID="af60239b0824673a94f249cc5c6c78634af4d57985275e7a98b0eadf1c360662" exitCode=7
Jul 16 22:17:41.337252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:41.336907    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerDied","Data":"af60239b0824673a94f249cc5c6c78634af4d57985275e7a98b0eadf1c360662"}
Jul 16 22:17:41.337252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:41.337120    2555 scope.go:117] "RemoveContainer" containerID="af60239b0824673a94f249cc5c6c78634af4d57985275e7a98b0eadf1c360662"
Jul 16 22:17:42.343494 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:42.343439    2555 generic.go:356] "Generic (PLEG): container finished" podID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerID="6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d" exitCode=7
Jul 16 22:17:42.343970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:42.343529    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerDied","Data":"6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d"}
Jul 16 22:17:42.343970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:42.343594    2555 scope.go:117] "RemoveContainer" containerID="af60239b0824673a94f249cc5c6c78634af4d57985275e7a98b0eadf1c360662"
Jul 16 22:17:42.343970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:42.343802    2555 scope.go:117] "RemoveContainer" containerID="6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d"
Jul 16 22:17:42.344130 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:17:42.344055    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk_odh-ai-gateway-infra(5696511b-079e-43eb-b50d-a2b16716e4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0"
Jul 16 22:17:43.350373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:43.350350    2555 scope.go:117] "RemoveContainer" containerID="6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d"
Jul 16 22:17:43.350778 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:17:43.350584    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk_odh-ai-gateway-infra(5696511b-079e-43eb-b50d-a2b16716e4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0"
Jul 16 22:17:52.285040 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:52.285006    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:17:54.670606 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:54.670581    2555 scope.go:117] "RemoveContainer" containerID="6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d"
Jul 16 22:17:55.417833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:55.417804    2555 generic.go:356] "Generic (PLEG): container finished" podID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerID="a020e5048f5aa38f0a96774596d55829fae2361ca4e04a18e4c55127c9d22326" exitCode=6
Jul 16 22:17:55.418019 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:55.417880    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerDied","Data":"a020e5048f5aa38f0a96774596d55829fae2361ca4e04a18e4c55127c9d22326"}
Jul 16 22:17:55.418019 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:55.417947    2555 scope.go:117] "RemoveContainer" containerID="6adadc154f050f386696a453998bbe9fad9c3512082a99abbd7ca87a51c00c1d"
Jul 16 22:17:55.418231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:55.418214    2555 scope.go:117] "RemoveContainer" containerID="a020e5048f5aa38f0a96774596d55829fae2361ca4e04a18e4c55127c9d22326"
Jul 16 22:17:55.418454 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:17:55.418430    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk_odh-ai-gateway-infra(5696511b-079e-43eb-b50d-a2b16716e4f0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0"
Jul 16 22:17:56.450678 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:56.450638    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:56.588210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:56.588181    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:56.672266 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:56.672233    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca\") pod \"5696511b-079e-43eb-b50d-a2b16716e4f0\" (UID: \"5696511b-079e-43eb-b50d-a2b16716e4f0\") "
Jul 16 22:17:56.672592 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:56.672572    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5696511b-079e-43eb-b50d-a2b16716e4f0" (UID: "5696511b-079e-43eb-b50d-a2b16716e4f0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:17:56.773754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:56.773680    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5696511b-079e-43eb-b50d-a2b16716e4f0-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:17:57.431195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.431166    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"
Jul 16 22:17:57.431195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.431184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk" event={"ID":"5696511b-079e-43eb-b50d-a2b16716e4f0","Type":"ContainerDied","Data":"a74bdd16ae99b89391717cf9f817459f9b8cb750907e832cfa9b9600adc3828c"}
Jul 16 22:17:57.431492 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.431246    2555 scope.go:117] "RemoveContainer" containerID="a020e5048f5aa38f0a96774596d55829fae2361ca4e04a18e4c55127c9d22326"
Jul 16 22:17:57.452943 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.452916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:57.457950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.457927    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-6qqlk"]
Jul 16 22:17:57.682290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:17:57.682207    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" path="/var/lib/kubelet/pods/5696511b-079e-43eb-b50d-a2b16716e4f0/volumes"
Jul 16 22:18:06.482344 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:06.482262    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:18:08.117013 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.116979    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117361    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117379    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117392    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117402    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117423    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117432    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117495    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:18:08.117681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117505    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:18:08.117681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117598    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.117681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117605    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b4409e63-00b5-4b1d-91ce-76fd177a28ff" containerName="revoke-keys"
Jul 16 22:18:08.117681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.117613    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:08.121990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.121974    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:08.124564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.124541    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:08.124668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.124588    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:18:08.128021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.127997    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:08.275830 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.275796    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q\" (UID: \"11bf2006-2931-4247-a384-58f34dc3c2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:08.377068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.376987    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q\" (UID: \"11bf2006-2931-4247-a384-58f34dc3c2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:08.377613 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.377597    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q\" (UID: \"11bf2006-2931-4247-a384-58f34dc3c2c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:08.433055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.433026    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:08.566827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:08.566804    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:08.568892 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:18:08.568864    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11bf2006_2931_4247_a384_58f34dc3c2c4.slice/crio-a995054cdf2e1fe975f906eb3f5e97b03dba244ffd298ebc0e8cf6d0b7e202ce WatchSource:0}: Error finding container a995054cdf2e1fe975f906eb3f5e97b03dba244ffd298ebc0e8cf6d0b7e202ce: Status 404 returned error can't find the container with id a995054cdf2e1fe975f906eb3f5e97b03dba244ffd298ebc0e8cf6d0b7e202ce
Jul 16 22:18:09.499455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:09.499369    2555 generic.go:356] "Generic (PLEG): container finished" podID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerID="39c4f850e4b5edb05562140bbeb9d7c44ac469ccb571fbdf69231c31fc91f66f" exitCode=7
Jul 16 22:18:09.499455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:09.499423    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" event={"ID":"11bf2006-2931-4247-a384-58f34dc3c2c4","Type":"ContainerDied","Data":"39c4f850e4b5edb05562140bbeb9d7c44ac469ccb571fbdf69231c31fc91f66f"}
Jul 16 22:18:09.499917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:09.499490    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" event={"ID":"11bf2006-2931-4247-a384-58f34dc3c2c4","Type":"ContainerStarted","Data":"a995054cdf2e1fe975f906eb3f5e97b03dba244ffd298ebc0e8cf6d0b7e202ce"}
Jul 16 22:18:09.499917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:09.499690    2555 scope.go:117] "RemoveContainer" containerID="39c4f850e4b5edb05562140bbeb9d7c44ac469ccb571fbdf69231c31fc91f66f"
Jul 16 22:18:10.506014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:10.505933    2555 generic.go:356] "Generic (PLEG): container finished" podID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerID="8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab" exitCode=7
Jul 16 22:18:10.506529 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:10.506015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" event={"ID":"11bf2006-2931-4247-a384-58f34dc3c2c4","Type":"ContainerDied","Data":"8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab"}
Jul 16 22:18:10.506529 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:10.506073    2555 scope.go:117] "RemoveContainer" containerID="39c4f850e4b5edb05562140bbeb9d7c44ac469ccb571fbdf69231c31fc91f66f"
Jul 16 22:18:10.506529 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:10.506239    2555 scope.go:117] "RemoveContainer" containerID="8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab"
Jul 16 22:18:10.506529 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:10.506457    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q_odh-ai-gateway-infra(11bf2006-2931-4247-a384-58f34dc3c2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4"
Jul 16 22:18:11.513051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:11.513017    2555 scope.go:117] "RemoveContainer" containerID="8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab"
Jul 16 22:18:11.513552 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:11.513267    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q_odh-ai-gateway-infra(11bf2006-2931-4247-a384-58f34dc3c2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4"
Jul 16 22:18:23.671185 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:23.671149    2555 scope.go:117] "RemoveContainer" containerID="8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab"
Jul 16 22:18:24.579869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:24.579786    2555 generic.go:356] "Generic (PLEG): container finished" podID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerID="be6b7750f0d5efff38f8a8370d46b08fdc7387410de9efd3ce6610a1af7e83e9" exitCode=7
Jul 16 22:18:24.580029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:24.579860    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" event={"ID":"11bf2006-2931-4247-a384-58f34dc3c2c4","Type":"ContainerDied","Data":"be6b7750f0d5efff38f8a8370d46b08fdc7387410de9efd3ce6610a1af7e83e9"}
Jul 16 22:18:24.580029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:24.579919    2555 scope.go:117] "RemoveContainer" containerID="8baf6fe2fee3e57b10a001e489a8079616a2a9310785c5af86f3518f7e86cfab"
Jul 16 22:18:24.580296 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:24.580282    2555 scope.go:117] "RemoveContainer" containerID="be6b7750f0d5efff38f8a8370d46b08fdc7387410de9efd3ce6610a1af7e83e9"
Jul 16 22:18:24.580519 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:24.580498    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q_odh-ai-gateway-infra(11bf2006-2931-4247-a384-58f34dc3c2c4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4"
Jul 16 22:18:25.609809 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:25.609775    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:25.751791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:25.751768    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:25.826125 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:25.826093    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca\") pod \"11bf2006-2931-4247-a384-58f34dc3c2c4\" (UID: \"11bf2006-2931-4247-a384-58f34dc3c2c4\") "
Jul 16 22:18:25.826451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:25.826428    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11bf2006-2931-4247-a384-58f34dc3c2c4" (UID: "11bf2006-2931-4247-a384-58f34dc3c2c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:18:25.927527 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:25.927429    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11bf2006-2931-4247-a384-58f34dc3c2c4-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:18:26.594121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:26.594091    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"
Jul 16 22:18:26.594390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:26.594089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q" event={"ID":"11bf2006-2931-4247-a384-58f34dc3c2c4","Type":"ContainerDied","Data":"a995054cdf2e1fe975f906eb3f5e97b03dba244ffd298ebc0e8cf6d0b7e202ce"}
Jul 16 22:18:26.594390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:26.594231    2555 scope.go:117] "RemoveContainer" containerID="be6b7750f0d5efff38f8a8370d46b08fdc7387410de9efd3ce6610a1af7e83e9"
Jul 16 22:18:26.616027 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:26.616005    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:26.624023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:26.624001    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tnx6q"]
Jul 16 22:18:27.676432 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:27.676401    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" path="/var/lib/kubelet/pods/11bf2006-2931-4247-a384-58f34dc3c2c4/volumes"
Jul 16 22:18:29.400698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.400659    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:29.401211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401147    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401167    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401209    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401218    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401292    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401305    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401315    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:29.401403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.401323    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5696511b-079e-43eb-b50d-a2b16716e4f0" containerName="revoke-keys"
Jul 16 22:18:29.405894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.405878    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:29.409325 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.409301    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:18:29.409497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.409330    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:29.412100 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.412080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:29.462940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.462893    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m\" (UID: \"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:29.564287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.564250    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m\" (UID: \"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:29.564915 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.564893    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m\" (UID: \"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:29.718167 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.718073    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:29.855494 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:29.855443    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:29.858977 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:18:29.858949    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fe52ed_1f8d_4f00_8ef3_0e910c90ebc4.slice/crio-43d3bb0b297499606cc9159eb9d9582d469d73f58264d7f9cf49ba628e26d36d WatchSource:0}: Error finding container 43d3bb0b297499606cc9159eb9d9582d469d73f58264d7f9cf49ba628e26d36d: Status 404 returned error can't find the container with id 43d3bb0b297499606cc9159eb9d9582d469d73f58264d7f9cf49ba628e26d36d
Jul 16 22:18:30.616302 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:30.616261    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerID="ff9ce10ecdd65c6c534eb77152d5eb20f47889b63b9366aa7583b5be5e32c77a" exitCode=6
Jul 16 22:18:30.616722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:30.616345    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" event={"ID":"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4","Type":"ContainerDied","Data":"ff9ce10ecdd65c6c534eb77152d5eb20f47889b63b9366aa7583b5be5e32c77a"}
Jul 16 22:18:30.616722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:30.616393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" event={"ID":"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4","Type":"ContainerStarted","Data":"43d3bb0b297499606cc9159eb9d9582d469d73f58264d7f9cf49ba628e26d36d"}
Jul 16 22:18:30.616722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:30.616584    2555 scope.go:117] "RemoveContainer" containerID="ff9ce10ecdd65c6c534eb77152d5eb20f47889b63b9366aa7583b5be5e32c77a"
Jul 16 22:18:31.624752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:31.624664    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerID="449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739" exitCode=6
Jul 16 22:18:31.625175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:31.624754    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" event={"ID":"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4","Type":"ContainerDied","Data":"449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739"}
Jul 16 22:18:31.625175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:31.624816    2555 scope.go:117] "RemoveContainer" containerID="ff9ce10ecdd65c6c534eb77152d5eb20f47889b63b9366aa7583b5be5e32c77a"
Jul 16 22:18:31.625175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:31.625015    2555 scope.go:117] "RemoveContainer" containerID="449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739"
Jul 16 22:18:31.625325 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:31.625249    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m_odh-ai-gateway-infra(c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4"
Jul 16 22:18:32.631640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:32.631612    2555 scope.go:117] "RemoveContainer" containerID="449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739"
Jul 16 22:18:32.632013 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:32.631791    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m_odh-ai-gateway-infra(c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4"
Jul 16 22:18:43.670390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:43.670354    2555 scope.go:117] "RemoveContainer" containerID="449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739"
Jul 16 22:18:44.698524 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:44.698417    2555 generic.go:356] "Generic (PLEG): container finished" podID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerID="57fd8d8fb03bd722b2dd304447a9cba540d73bc4c3b5fe7bfa414650873f737c" exitCode=6
Jul 16 22:18:44.698940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:44.698522    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" event={"ID":"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4","Type":"ContainerDied","Data":"57fd8d8fb03bd722b2dd304447a9cba540d73bc4c3b5fe7bfa414650873f737c"}
Jul 16 22:18:44.698940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:44.698587    2555 scope.go:117] "RemoveContainer" containerID="449b5fc0db0a1b05ae8ad795caa3a6c7e4e1ae44ed82b82b6e76e9486007c739"
Jul 16 22:18:44.698940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:44.698936    2555 scope.go:117] "RemoveContainer" containerID="57fd8d8fb03bd722b2dd304447a9cba540d73bc4c3b5fe7bfa414650873f737c"
Jul 16 22:18:44.699158 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:18:44.699132    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m_odh-ai-gateway-infra(c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4"
Jul 16 22:18:45.724353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:45.724315    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:45.851686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:45.851660    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:46.007174 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.007082    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca\") pod \"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4\" (UID: \"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4\") "
Jul 16 22:18:46.007431 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.007405    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" (UID: "c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:18:46.108070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.108039    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:18:46.712683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.712645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m" event={"ID":"c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4","Type":"ContainerDied","Data":"43d3bb0b297499606cc9159eb9d9582d469d73f58264d7f9cf49ba628e26d36d"}
Jul 16 22:18:46.712875 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.712704    2555 scope.go:117] "RemoveContainer" containerID="57fd8d8fb03bd722b2dd304447a9cba540d73bc4c3b5fe7bfa414650873f737c"
Jul 16 22:18:46.712875 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.712659    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"
Jul 16 22:18:46.735613 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.735581    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:46.742005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:46.741985    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-92t7m"]
Jul 16 22:18:47.676442 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:47.676408    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" path="/var/lib/kubelet/pods/c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4/volumes"
Jul 16 22:18:58.206665 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206584    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206932    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206943    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206959    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206964    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206971    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.206976    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="11bf2006-2931-4247-a384-58f34dc3c2c4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.207029    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.207036    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.207131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.207045    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:18:58.211703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.211686    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:18:58.214791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.214768    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:18:58.215265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.215253    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:18:58.239176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.239151    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:18:58.318437 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.318405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl\" (UID: \"4b6961f0-3491-420e-ba12-aa5fb4e4d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:18:58.419042 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.419002    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl\" (UID: \"4b6961f0-3491-420e-ba12-aa5fb4e4d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:18:58.419718 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.419693    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl\" (UID: \"4b6961f0-3491-420e-ba12-aa5fb4e4d507\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:18:58.521582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.521489    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:18:58.676219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.676189    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:18:58.677436 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:18:58.677412    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b6961f0_3491_420e_ba12_aa5fb4e4d507.slice/crio-fbab839bc072b55265e24cda2925d2008a183b1c0b9ee0b929e98623edaffc7c WatchSource:0}: Error finding container fbab839bc072b55265e24cda2925d2008a183b1c0b9ee0b929e98623edaffc7c: Status 404 returned error can't find the container with id fbab839bc072b55265e24cda2925d2008a183b1c0b9ee0b929e98623edaffc7c
Jul 16 22:18:58.775518 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:58.775419    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" event={"ID":"4b6961f0-3491-420e-ba12-aa5fb4e4d507","Type":"ContainerStarted","Data":"fbab839bc072b55265e24cda2925d2008a183b1c0b9ee0b929e98623edaffc7c"}
Jul 16 22:18:59.781686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:59.781653    2555 generic.go:356] "Generic (PLEG): container finished" podID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerID="93a1eac7b185d2ca05654a69301cdd5b95cbec5714ade38a4321856462f7e5dd" exitCode=7
Jul 16 22:18:59.782110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:59.781743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" event={"ID":"4b6961f0-3491-420e-ba12-aa5fb4e4d507","Type":"ContainerDied","Data":"93a1eac7b185d2ca05654a69301cdd5b95cbec5714ade38a4321856462f7e5dd"}
Jul 16 22:18:59.782110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:18:59.781877    2555 scope.go:117] "RemoveContainer" containerID="93a1eac7b185d2ca05654a69301cdd5b95cbec5714ade38a4321856462f7e5dd"
Jul 16 22:19:00.788979 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:00.788941    2555 generic.go:356] "Generic (PLEG): container finished" podID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerID="8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e" exitCode=7
Jul 16 22:19:00.789388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:00.789032    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" event={"ID":"4b6961f0-3491-420e-ba12-aa5fb4e4d507","Type":"ContainerDied","Data":"8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e"}
Jul 16 22:19:00.789388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:00.789127    2555 scope.go:117] "RemoveContainer" containerID="93a1eac7b185d2ca05654a69301cdd5b95cbec5714ade38a4321856462f7e5dd"
Jul 16 22:19:00.789388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:00.789319    2555 scope.go:117] "RemoveContainer" containerID="8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e"
Jul 16 22:19:00.789572 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:00.789549    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl_odh-ai-gateway-infra(4b6961f0-3491-420e-ba12-aa5fb4e4d507)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507"
Jul 16 22:19:01.796412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:01.796387    2555 scope.go:117] "RemoveContainer" containerID="8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e"
Jul 16 22:19:01.796813 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:01.796600    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl_odh-ai-gateway-infra(4b6961f0-3491-420e-ba12-aa5fb4e4d507)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507"
Jul 16 22:19:11.208219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:11.208180    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:19:13.670766 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:13.670738    2555 scope.go:117] "RemoveContainer" containerID="8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e"
Jul 16 22:19:14.866444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:14.866412    2555 generic.go:356] "Generic (PLEG): container finished" podID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerID="20d9b78b914a5cfeaa911ffe2e904212c92b5c3a57a4c90abd271deb0dc4ef55" exitCode=7
Jul 16 22:19:14.866919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:14.866499    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" event={"ID":"4b6961f0-3491-420e-ba12-aa5fb4e4d507","Type":"ContainerDied","Data":"20d9b78b914a5cfeaa911ffe2e904212c92b5c3a57a4c90abd271deb0dc4ef55"}
Jul 16 22:19:14.866919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:14.866574    2555 scope.go:117] "RemoveContainer" containerID="8a55303d212a2056ed417f8860790348845f9889dbbeafc98b00ef6f6f9e4b0e"
Jul 16 22:19:14.866919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:14.866915    2555 scope.go:117] "RemoveContainer" containerID="20d9b78b914a5cfeaa911ffe2e904212c92b5c3a57a4c90abd271deb0dc4ef55"
Jul 16 22:19:14.867144 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:14.867121    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl_odh-ai-gateway-infra(4b6961f0-3491-420e-ba12-aa5fb4e4d507)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507"
Jul 16 22:19:15.901605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:15.901573    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:19:16.041761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.041732    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:19:16.074749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.074678    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca\") pod \"4b6961f0-3491-420e-ba12-aa5fb4e4d507\" (UID: \"4b6961f0-3491-420e-ba12-aa5fb4e4d507\") "
Jul 16 22:19:16.075002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.074982    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4b6961f0-3491-420e-ba12-aa5fb4e4d507" (UID: "4b6961f0-3491-420e-ba12-aa5fb4e4d507"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:16.176163 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.176126    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4b6961f0-3491-420e-ba12-aa5fb4e4d507-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:19:16.880567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.880534    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"
Jul 16 22:19:16.880567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.880549    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl" event={"ID":"4b6961f0-3491-420e-ba12-aa5fb4e4d507","Type":"ContainerDied","Data":"fbab839bc072b55265e24cda2925d2008a183b1c0b9ee0b929e98623edaffc7c"}
Jul 16 22:19:16.880791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.880643    2555 scope.go:117] "RemoveContainer" containerID="20d9b78b914a5cfeaa911ffe2e904212c92b5c3a57a4c90abd271deb0dc4ef55"
Jul 16 22:19:16.911927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.911903    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:19:16.917848 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:16.917824    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9ksvl"]
Jul 16 22:19:17.675851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:17.675818    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" path="/var/lib/kubelet/pods/4b6961f0-3491-420e-ba12-aa5fb4e4d507/volumes"
Jul 16 22:19:19.482632 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.482602    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:19.483014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.482959    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:19:19.483014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.482971    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6fe52ed-1f8d-4f00-8ef3-0e910c90ebc4" containerName="revoke-keys"
Jul 16 22:19:19.483014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.482985    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.482991    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.483008    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.483018    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.483076    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.483087    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.483179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.483099    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:19.487329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.487309    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:19.493982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.493951    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:19:19.493982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.493951    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:19:19.510736 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.510704    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:19.603703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.603668    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f\" (UID: \"71e1c2fe-cf83-4519-9ee1-3287f16fe694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:19.705070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.705036    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f\" (UID: \"71e1c2fe-cf83-4519-9ee1-3287f16fe694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:19.705728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.705709    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f\" (UID: \"71e1c2fe-cf83-4519-9ee1-3287f16fe694\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:19.797694 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.797666    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:19.963440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:19.963319    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:19.965724 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:19:19.965698    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71e1c2fe_cf83_4519_9ee1_3287f16fe694.slice/crio-19c5e64496a2398f68f37d729e793e1d68ca8b1933794b3e8b573976f5441896 WatchSource:0}: Error finding container 19c5e64496a2398f68f37d729e793e1d68ca8b1933794b3e8b573976f5441896: Status 404 returned error can't find the container with id 19c5e64496a2398f68f37d729e793e1d68ca8b1933794b3e8b573976f5441896
Jul 16 22:19:20.904397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:20.904364    2555 generic.go:356] "Generic (PLEG): container finished" podID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerID="b93dded3f61568d2b302005ddfbf2636b40e7bade8d94c485806ac95550857a7" exitCode=6
Jul 16 22:19:20.904817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:20.904407    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerDied","Data":"b93dded3f61568d2b302005ddfbf2636b40e7bade8d94c485806ac95550857a7"}
Jul 16 22:19:20.904817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:20.904444    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerStarted","Data":"19c5e64496a2398f68f37d729e793e1d68ca8b1933794b3e8b573976f5441896"}
Jul 16 22:19:20.904817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:20.904633    2555 scope.go:117] "RemoveContainer" containerID="b93dded3f61568d2b302005ddfbf2636b40e7bade8d94c485806ac95550857a7"
Jul 16 22:19:21.910884 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:21.910848    2555 generic.go:356] "Generic (PLEG): container finished" podID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerID="e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43" exitCode=6
Jul 16 22:19:21.911376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:21.910945    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerDied","Data":"e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43"}
Jul 16 22:19:21.911376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:21.911060    2555 scope.go:117] "RemoveContainer" containerID="b93dded3f61568d2b302005ddfbf2636b40e7bade8d94c485806ac95550857a7"
Jul 16 22:19:21.911376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:21.911123    2555 scope.go:117] "RemoveContainer" containerID="e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43"
Jul 16 22:19:21.911376 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:21.911335    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f_odh-ai-gateway-infra(71e1c2fe-cf83-4519-9ee1-3287f16fe694)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694"
Jul 16 22:19:22.917885 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:22.917854    2555 scope.go:117] "RemoveContainer" containerID="e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43"
Jul 16 22:19:22.918251 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:22.918057    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f_odh-ai-gateway-infra(71e1c2fe-cf83-4519-9ee1-3287f16fe694)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694"
Jul 16 22:19:28.157379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:28.157338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:19:30.016294 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.016262    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:30.016799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.016622    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:30.016799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.016634    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b6961f0-3491-420e-ba12-aa5fb4e4d507" containerName="revoke-keys"
Jul 16 22:19:30.019635 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.019618    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:30.047086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.047052    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:30.095892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.095854    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6lpz6\" (UID: \"96d43bb6-5dec-49fc-a260-14726595c71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:30.196779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.196748    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6lpz6\" (UID: \"96d43bb6-5dec-49fc-a260-14726595c71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:30.197395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.197375    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6lpz6\" (UID: \"96d43bb6-5dec-49fc-a260-14726595c71d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:30.328879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.328850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:30.700309 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.700284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:30.702422 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:19:30.702383    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96d43bb6_5dec_49fc_a260_14726595c71d.slice/crio-57d011cf783b0e5ed4e370cfaae7594c27df861b1fb345af18abb75a5a64831a WatchSource:0}: Error finding container 57d011cf783b0e5ed4e370cfaae7594c27df861b1fb345af18abb75a5a64831a: Status 404 returned error can't find the container with id 57d011cf783b0e5ed4e370cfaae7594c27df861b1fb345af18abb75a5a64831a
Jul 16 22:19:30.959252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:30.959174    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerStarted","Data":"57d011cf783b0e5ed4e370cfaae7594c27df861b1fb345af18abb75a5a64831a"}
Jul 16 22:19:31.965410 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:31.965375    2555 generic.go:356] "Generic (PLEG): container finished" podID="96d43bb6-5dec-49fc-a260-14726595c71d" containerID="7b8911de7edf536c44b06cce65502958e09fcb25089bfb0ab16ed07a3d9c1b2e" exitCode=7
Jul 16 22:19:31.965862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:31.965484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerDied","Data":"7b8911de7edf536c44b06cce65502958e09fcb25089bfb0ab16ed07a3d9c1b2e"}
Jul 16 22:19:31.965862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:31.965675    2555 scope.go:117] "RemoveContainer" containerID="7b8911de7edf536c44b06cce65502958e09fcb25089bfb0ab16ed07a3d9c1b2e"
Jul 16 22:19:32.973755 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:32.973723    2555 generic.go:356] "Generic (PLEG): container finished" podID="96d43bb6-5dec-49fc-a260-14726595c71d" containerID="a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58" exitCode=7
Jul 16 22:19:32.974250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:32.973815    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerDied","Data":"a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58"}
Jul 16 22:19:32.974250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:32.973883    2555 scope.go:117] "RemoveContainer" containerID="7b8911de7edf536c44b06cce65502958e09fcb25089bfb0ab16ed07a3d9c1b2e"
Jul 16 22:19:32.974250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:32.974143    2555 scope.go:117] "RemoveContainer" containerID="a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58"
Jul 16 22:19:32.974392 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:32.974377    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6lpz6_odh-ai-gateway-infra(96d43bb6-5dec-49fc-a260-14726595c71d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" podUID="96d43bb6-5dec-49fc-a260-14726595c71d"
Jul 16 22:19:33.980617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:33.980587    2555 scope.go:117] "RemoveContainer" containerID="a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58"
Jul 16 22:19:33.981012 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:33.980817    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6lpz6_odh-ai-gateway-infra(96d43bb6-5dec-49fc-a260-14726595c71d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" podUID="96d43bb6-5dec-49fc-a260-14726595c71d"
Jul 16 22:19:36.671281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:36.671256    2555 scope.go:117] "RemoveContainer" containerID="e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43"
Jul 16 22:19:36.998758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:36.998727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerStarted","Data":"bed4e536fa453dc3e0e759e3713ea0051156a7e0ddc839022e36a5f6691888d8"}
Jul 16 22:19:37.061045 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:37.060984    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" podStartSLOduration=18.060963418 podStartE2EDuration="18.060963418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:37.057583671 +0000 UTC m=+1831.894637463" watchObservedRunningTime="2026-07-16 22:19:37.060963418 +0000 UTC m=+1831.898017220"
Jul 16 22:19:38.006578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:38.006545    2555 generic.go:356] "Generic (PLEG): container finished" podID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerID="bed4e536fa453dc3e0e759e3713ea0051156a7e0ddc839022e36a5f6691888d8" exitCode=6
Jul 16 22:19:38.006977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:38.006620    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerDied","Data":"bed4e536fa453dc3e0e759e3713ea0051156a7e0ddc839022e36a5f6691888d8"}
Jul 16 22:19:38.006977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:38.006680    2555 scope.go:117] "RemoveContainer" containerID="e1813c33877089e89e09ed4c057fd162380fcfb6de03c7da98ad285083ca3e43"
Jul 16 22:19:38.007058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:38.007044    2555 scope.go:117] "RemoveContainer" containerID="bed4e536fa453dc3e0e759e3713ea0051156a7e0ddc839022e36a5f6691888d8"
Jul 16 22:19:38.007270 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:38.007252    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f_odh-ai-gateway-infra(71e1c2fe-cf83-4519-9ee1-3287f16fe694)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694"
Jul 16 22:19:38.074948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:38.074917    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:39.148218 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:39.148195    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:39.278248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:39.278215    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca\") pod \"71e1c2fe-cf83-4519-9ee1-3287f16fe694\" (UID: \"71e1c2fe-cf83-4519-9ee1-3287f16fe694\") "
Jul 16 22:19:39.278620 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:39.278598    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "71e1c2fe-cf83-4519-9ee1-3287f16fe694" (UID: "71e1c2fe-cf83-4519-9ee1-3287f16fe694"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:39.379854 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:39.379814    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/71e1c2fe-cf83-4519-9ee1-3287f16fe694-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:19:40.020041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:40.019947    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"
Jul 16 22:19:40.020041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:40.019967    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f" event={"ID":"71e1c2fe-cf83-4519-9ee1-3287f16fe694","Type":"ContainerDied","Data":"19c5e64496a2398f68f37d729e793e1d68ca8b1933794b3e8b573976f5441896"}
Jul 16 22:19:40.020262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:40.020051    2555 scope.go:117] "RemoveContainer" containerID="bed4e536fa453dc3e0e759e3713ea0051156a7e0ddc839022e36a5f6691888d8"
Jul 16 22:19:40.042882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:40.042848    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:40.050312 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:40.050276    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-58d5f"]
Jul 16 22:19:41.676913 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:41.676878    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" path="/var/lib/kubelet/pods/71e1c2fe-cf83-4519-9ee1-3287f16fe694/volumes"
Jul 16 22:19:45.673633 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:45.673599    2555 scope.go:117] "RemoveContainer" containerID="a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58"
Jul 16 22:19:46.052851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:46.052816    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerStarted","Data":"6f39b4c63a11362856ba99e5d33825830cc004369edeab1177a60e32f1d64c15"}
Jul 16 22:19:46.069555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:46.069507    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" podStartSLOduration=17.069491291 podStartE2EDuration="17.069491291s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:46.068562144 +0000 UTC m=+1840.905615957" watchObservedRunningTime="2026-07-16 22:19:46.069491291 +0000 UTC m=+1840.906545090"
Jul 16 22:19:47.059888 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:47.059853    2555 generic.go:356] "Generic (PLEG): container finished" podID="96d43bb6-5dec-49fc-a260-14726595c71d" containerID="6f39b4c63a11362856ba99e5d33825830cc004369edeab1177a60e32f1d64c15" exitCode=7
Jul 16 22:19:47.060335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:47.059929    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerDied","Data":"6f39b4c63a11362856ba99e5d33825830cc004369edeab1177a60e32f1d64c15"}
Jul 16 22:19:47.060335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:47.059985    2555 scope.go:117] "RemoveContainer" containerID="a18f9f07cd70b4624b3efacc589a5fb6de4d5741407f756903c4ca8b142a5b58"
Jul 16 22:19:47.060335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:47.060255    2555 scope.go:117] "RemoveContainer" containerID="6f39b4c63a11362856ba99e5d33825830cc004369edeab1177a60e32f1d64c15"
Jul 16 22:19:47.060560 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:47.060492    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6lpz6_odh-ai-gateway-infra(96d43bb6-5dec-49fc-a260-14726595c71d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" podUID="96d43bb6-5dec-49fc-a260-14726595c71d"
Jul 16 22:19:47.081619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:47.081591    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:48.206330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.206306    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:48.272599 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272567    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:19:48.272964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272952    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272975    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272981    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272989    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.272995    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273001    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273006    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273018    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273023    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273029    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273035    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273090    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273097    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273103    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.273252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.273109    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:19:48.276142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.276126    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:19:48.285779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.285754    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:19:48.363296 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.363204    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca\") pod \"96d43bb6-5dec-49fc-a260-14726595c71d\" (UID: \"96d43bb6-5dec-49fc-a260-14726595c71d\") "
Jul 16 22:19:48.363502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.363450    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv\" (UID: \"aee17391-8d6d-4474-9f4c-bc895bd753ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:19:48.363642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.363624    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "96d43bb6-5dec-49fc-a260-14726595c71d" (UID: "96d43bb6-5dec-49fc-a260-14726595c71d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:19:48.464301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.464254    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv\" (UID: \"aee17391-8d6d-4474-9f4c-bc895bd753ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:19:48.464301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.464312    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/96d43bb6-5dec-49fc-a260-14726595c71d-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:19:48.464911 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.464891    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv\" (UID: \"aee17391-8d6d-4474-9f4c-bc895bd753ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:19:48.587644 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.587609    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:19:48.732287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:48.732258    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:19:48.733192 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:19:48.733170    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaee17391_8d6d_4474_9f4c_bc895bd753ad.slice/crio-fbc72e62f4977af2df6c26cde0ea39631aa3f25d5328c3ccb625835439e25b8e WatchSource:0}: Error finding container fbc72e62f4977af2df6c26cde0ea39631aa3f25d5328c3ccb625835439e25b8e: Status 404 returned error can't find the container with id fbc72e62f4977af2df6c26cde0ea39631aa3f25d5328c3ccb625835439e25b8e
Jul 16 22:19:49.074348 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.074318    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"
Jul 16 22:19:49.074348 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.074332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6" event={"ID":"96d43bb6-5dec-49fc-a260-14726595c71d","Type":"ContainerDied","Data":"57d011cf783b0e5ed4e370cfaae7594c27df861b1fb345af18abb75a5a64831a"}
Jul 16 22:19:49.074644 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.074397    2555 scope.go:117] "RemoveContainer" containerID="6f39b4c63a11362856ba99e5d33825830cc004369edeab1177a60e32f1d64c15"
Jul 16 22:19:49.076201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.076177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerStarted","Data":"e1a92637dc8a4c5e75c9b41904ec00bbc836c546c211e0e71dd42d77928910a4"}
Jul 16 22:19:49.076300 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.076227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerStarted","Data":"fbc72e62f4977af2df6c26cde0ea39631aa3f25d5328c3ccb625835439e25b8e"}
Jul 16 22:19:49.130097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.130040    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" podStartSLOduration=1.130022881 podStartE2EDuration="1.130022881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:19:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:19:49.121841364 +0000 UTC m=+1843.958895163" watchObservedRunningTime="2026-07-16 22:19:49.130022881 +0000 UTC m=+1843.967076717"
Jul 16 22:19:49.160592 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.155161    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:49.163477 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.163436    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6lpz6"]
Jul 16 22:19:49.676806 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:49.676729    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" path="/var/lib/kubelet/pods/96d43bb6-5dec-49fc-a260-14726595c71d/volumes"
Jul 16 22:19:50.083384 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:50.083349    2555 generic.go:356] "Generic (PLEG): container finished" podID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerID="e1a92637dc8a4c5e75c9b41904ec00bbc836c546c211e0e71dd42d77928910a4" exitCode=6
Jul 16 22:19:50.083615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:50.083433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerDied","Data":"e1a92637dc8a4c5e75c9b41904ec00bbc836c546c211e0e71dd42d77928910a4"}
Jul 16 22:19:50.083714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:50.083698    2555 scope.go:117] "RemoveContainer" containerID="e1a92637dc8a4c5e75c9b41904ec00bbc836c546c211e0e71dd42d77928910a4"
Jul 16 22:19:51.090889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:51.090848    2555 generic.go:356] "Generic (PLEG): container finished" podID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerID="30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506" exitCode=6
Jul 16 22:19:51.091297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:51.090927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerDied","Data":"30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506"}
Jul 16 22:19:51.091297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:51.091013    2555 scope.go:117] "RemoveContainer" containerID="e1a92637dc8a4c5e75c9b41904ec00bbc836c546c211e0e71dd42d77928910a4"
Jul 16 22:19:51.091297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:51.091255    2555 scope.go:117] "RemoveContainer" containerID="30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506"
Jul 16 22:19:51.091489 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:51.091451    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv_odh-ai-gateway-infra(aee17391-8d6d-4474-9f4c-bc895bd753ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad"
Jul 16 22:19:52.097603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:19:52.097576    2555 scope.go:117] "RemoveContainer" containerID="30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506"
Jul 16 22:19:52.097975 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:19:52.097759    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv_odh-ai-gateway-infra(aee17391-8d6d-4474-9f4c-bc895bd753ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad"
Jul 16 22:20:06.670672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:06.670634    2555 scope.go:117] "RemoveContainer" containerID="30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506"
Jul 16 22:20:07.175261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:07.175228    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerStarted","Data":"4916d74a97f17d634f1291ab6cd83630974549fa9a74e2e74ae14c71e941eec6"}
Jul 16 22:20:08.187655 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:08.187615    2555 generic.go:356] "Generic (PLEG): container finished" podID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerID="4916d74a97f17d634f1291ab6cd83630974549fa9a74e2e74ae14c71e941eec6" exitCode=6
Jul 16 22:20:08.188066 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:08.187681    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerDied","Data":"4916d74a97f17d634f1291ab6cd83630974549fa9a74e2e74ae14c71e941eec6"}
Jul 16 22:20:08.188066 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:08.187765    2555 scope.go:117] "RemoveContainer" containerID="30ee9f6b6a5b09def66a5ccfa702b76505f2fa48f9b492b8578240fbf34e7506"
Jul 16 22:20:08.188066 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:08.188002    2555 scope.go:117] "RemoveContainer" containerID="4916d74a97f17d634f1291ab6cd83630974549fa9a74e2e74ae14c71e941eec6"
Jul 16 22:20:08.188240 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:08.188222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv_odh-ai-gateway-infra(aee17391-8d6d-4474-9f4c-bc895bd753ad)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad"
Jul 16 22:20:08.201412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:08.201389    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:20:09.329827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:09.329800    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:20:09.443865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:09.443825    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca\") pod \"aee17391-8d6d-4474-9f4c-bc895bd753ad\" (UID: \"aee17391-8d6d-4474-9f4c-bc895bd753ad\") "
Jul 16 22:20:09.444189 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:09.444165    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aee17391-8d6d-4474-9f4c-bc895bd753ad" (UID: "aee17391-8d6d-4474-9f4c-bc895bd753ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:09.544794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:09.544767    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aee17391-8d6d-4474-9f4c-bc895bd753ad-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:20:10.201553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:10.201520    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"
Jul 16 22:20:10.201553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:10.201548    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv" event={"ID":"aee17391-8d6d-4474-9f4c-bc895bd753ad","Type":"ContainerDied","Data":"fbc72e62f4977af2df6c26cde0ea39631aa3f25d5328c3ccb625835439e25b8e"}
Jul 16 22:20:10.201781 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:10.201609    2555 scope.go:117] "RemoveContainer" containerID="4916d74a97f17d634f1291ab6cd83630974549fa9a74e2e74ae14c71e941eec6"
Jul 16 22:20:10.218006 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:10.217980    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:20:10.221188 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:10.221167    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wvmtv"]
Jul 16 22:20:11.676081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:11.676046    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" path="/var/lib/kubelet/pods/aee17391-8d6d-4474-9f4c-bc895bd753ad/volumes"
Jul 16 22:20:19.557596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557558    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557892    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557904    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557915    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557920    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557926    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.557933    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.558001    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.558009    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.558016    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="96d43bb6-5dec-49fc-a260-14726595c71d" containerName="revoke-keys"
Jul 16 22:20:19.558069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.558022    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="71e1c2fe-cf83-4519-9ee1-3287f16fe694" containerName="revoke-keys"
Jul 16 22:20:19.562183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.562166    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:19.564980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.564956    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:20:19.565104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.565001    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:20:19.572116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.572086    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:19.624444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.624414    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl\" (UID: \"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:19.725017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.724979    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl\" (UID: \"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:19.725648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.725629    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl\" (UID: \"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:19.873298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:19.873221    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:20.010337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.010311    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:20.012309 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:20:20.012269    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bc21ddd_1a87_48ee_8ae1_1322dfa478ac.slice/crio-9f2eeac1039319cc3ab78493a92ebb326e5da38581e235ec580e399a4b4e73a7 WatchSource:0}: Error finding container 9f2eeac1039319cc3ab78493a92ebb326e5da38581e235ec580e399a4b4e73a7: Status 404 returned error can't find the container with id 9f2eeac1039319cc3ab78493a92ebb326e5da38581e235ec580e399a4b4e73a7
Jul 16 22:20:20.065078 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.065049    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:20.065493 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.065479    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aee17391-8d6d-4474-9f4c-bc895bd753ad" containerName="revoke-keys"
Jul 16 22:20:20.069500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.069484    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:20.076750 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.076715    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:20.129404 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.129326    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-x6qz8\" (UID: \"ff162e5c-31bc-4a4a-ada4-008b2f358596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:20.235705 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.230723    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-x6qz8\" (UID: \"ff162e5c-31bc-4a4a-ada4-008b2f358596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:20.235705 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.231601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-x6qz8\" (UID: \"ff162e5c-31bc-4a4a-ada4-008b2f358596\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:20.252518 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.252482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerStarted","Data":"9f2eeac1039319cc3ab78493a92ebb326e5da38581e235ec580e399a4b4e73a7"}
Jul 16 22:20:20.382284 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.382202    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:20.516211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:20.516188    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:20.517682 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:20:20.517652    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff162e5c_31bc_4a4a_ada4_008b2f358596.slice/crio-8784c51621adbf4595c86a176e850ac8de5c1ed12f30de470afa54a9ef925eab WatchSource:0}: Error finding container 8784c51621adbf4595c86a176e850ac8de5c1ed12f30de470afa54a9ef925eab: Status 404 returned error can't find the container with id 8784c51621adbf4595c86a176e850ac8de5c1ed12f30de470afa54a9ef925eab
Jul 16 22:20:21.258873 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.258843    2555 generic.go:356] "Generic (PLEG): container finished" podID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerID="9e4c9cafc3657b529bc39a6f23dfec56da2dd461cfb61583f6ac2cbf6bbbc68c" exitCode=6
Jul 16 22:20:21.259316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.258906    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerDied","Data":"9e4c9cafc3657b529bc39a6f23dfec56da2dd461cfb61583f6ac2cbf6bbbc68c"}
Jul 16 22:20:21.259316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.259130    2555 scope.go:117] "RemoveContainer" containerID="9e4c9cafc3657b529bc39a6f23dfec56da2dd461cfb61583f6ac2cbf6bbbc68c"
Jul 16 22:20:21.260605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.260583    2555 generic.go:356] "Generic (PLEG): container finished" podID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerID="4d05f03dd87c528ef42226003b081937131d7c4c7e02a4a77d7905801ce88d85" exitCode=7
Jul 16 22:20:21.260704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.260630    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerDied","Data":"4d05f03dd87c528ef42226003b081937131d7c4c7e02a4a77d7905801ce88d85"}
Jul 16 22:20:21.260704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.260663    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerStarted","Data":"8784c51621adbf4595c86a176e850ac8de5c1ed12f30de470afa54a9ef925eab"}
Jul 16 22:20:21.260825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:21.260814    2555 scope.go:117] "RemoveContainer" containerID="4d05f03dd87c528ef42226003b081937131d7c4c7e02a4a77d7905801ce88d85"
Jul 16 22:20:22.268144 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.268047    2555 generic.go:356] "Generic (PLEG): container finished" podID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerID="0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625" exitCode=6
Jul 16 22:20:22.268144 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.268126    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerDied","Data":"0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625"}
Jul 16 22:20:22.268667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.268191    2555 scope.go:117] "RemoveContainer" containerID="9e4c9cafc3657b529bc39a6f23dfec56da2dd461cfb61583f6ac2cbf6bbbc68c"
Jul 16 22:20:22.268667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.268367    2555 scope.go:117] "RemoveContainer" containerID="0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625"
Jul 16 22:20:22.268667 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:22.268634    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl_odh-ai-gateway-infra(1bc21ddd-1a87-48ee-8ae1-1322dfa478ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac"
Jul 16 22:20:22.270050 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.270030    2555 generic.go:356] "Generic (PLEG): container finished" podID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerID="ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb" exitCode=7
Jul 16 22:20:22.270149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.270087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerDied","Data":"ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb"}
Jul 16 22:20:22.270281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.270268    2555 scope.go:117] "RemoveContainer" containerID="ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb"
Jul 16 22:20:22.270441 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:22.270427    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-x6qz8_odh-ai-gateway-infra(ff162e5c-31bc-4a4a-ada4-008b2f358596)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596"
Jul 16 22:20:22.279770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:22.279744    2555 scope.go:117] "RemoveContainer" containerID="4d05f03dd87c528ef42226003b081937131d7c4c7e02a4a77d7905801ce88d85"
Jul 16 22:20:23.277252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:23.277221    2555 scope.go:117] "RemoveContainer" containerID="0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625"
Jul 16 22:20:23.277693 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:23.277498    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl_odh-ai-gateway-infra(1bc21ddd-1a87-48ee-8ae1-1322dfa478ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac"
Jul 16 22:20:23.278938 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:23.278914    2555 scope.go:117] "RemoveContainer" containerID="ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb"
Jul 16 22:20:23.279126 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:23.279108    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-x6qz8_odh-ai-gateway-infra(ff162e5c-31bc-4a4a-ada4-008b2f358596)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596"
Jul 16 22:20:33.675899 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:33.675828    2555 scope.go:117] "RemoveContainer" containerID="0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625"
Jul 16 22:20:33.676287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:33.676025    2555 scope.go:117] "RemoveContainer" containerID="ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb"
Jul 16 22:20:34.336188 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:34.336154    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerStarted","Data":"8d069920273c6b7b4f6f4818fe1027d4bcb83377e1cccdba1e7ef2c3cdf320a4"}
Jul 16 22:20:34.338072 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:34.338044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerStarted","Data":"cddc9d49c88f5ad72cd4c72cf298adf47c77cefda0c30648f981615c4df382d2"}
Jul 16 22:20:34.354640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:34.354593    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" podStartSLOduration=14.354553578 podStartE2EDuration="14.354553578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:34.351954763 +0000 UTC m=+1889.189008578" watchObservedRunningTime="2026-07-16 22:20:34.354553578 +0000 UTC m=+1889.191607378"
Jul 16 22:20:34.368161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:34.368110    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" podStartSLOduration=15.368086417 podStartE2EDuration="15.368086417s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:20:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:20:34.366994027 +0000 UTC m=+1889.204047840" watchObservedRunningTime="2026-07-16 22:20:34.368086417 +0000 UTC m=+1889.205140217"
Jul 16 22:20:35.344769 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.344727    2555 generic.go:356] "Generic (PLEG): container finished" podID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerID="cddc9d49c88f5ad72cd4c72cf298adf47c77cefda0c30648f981615c4df382d2" exitCode=6
Jul 16 22:20:35.345211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.344807    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerDied","Data":"cddc9d49c88f5ad72cd4c72cf298adf47c77cefda0c30648f981615c4df382d2"}
Jul 16 22:20:35.345211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.344861    2555 scope.go:117] "RemoveContainer" containerID="0288178ca5aac084ca27e5c9efd73656c999346b80ba2989ab7ced6ac4d64625"
Jul 16 22:20:35.345320 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.345208    2555 scope.go:117] "RemoveContainer" containerID="cddc9d49c88f5ad72cd4c72cf298adf47c77cefda0c30648f981615c4df382d2"
Jul 16 22:20:35.345509 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:35.345484    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl_odh-ai-gateway-infra(1bc21ddd-1a87-48ee-8ae1-1322dfa478ac)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac"
Jul 16 22:20:35.346750 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.346729    2555 generic.go:356] "Generic (PLEG): container finished" podID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerID="8d069920273c6b7b4f6f4818fe1027d4bcb83377e1cccdba1e7ef2c3cdf320a4" exitCode=7
Jul 16 22:20:35.346836 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.346797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerDied","Data":"8d069920273c6b7b4f6f4818fe1027d4bcb83377e1cccdba1e7ef2c3cdf320a4"}
Jul 16 22:20:35.347033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.347021    2555 scope.go:117] "RemoveContainer" containerID="8d069920273c6b7b4f6f4818fe1027d4bcb83377e1cccdba1e7ef2c3cdf320a4"
Jul 16 22:20:35.347189 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:35.347174    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-x6qz8_odh-ai-gateway-infra(ff162e5c-31bc-4a4a-ada4-008b2f358596)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596"
Jul 16 22:20:35.356355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.356333    2555 scope.go:117] "RemoveContainer" containerID="ba45f91d23f59e60e0de6870f96ad3c22785ddded077a6c64f9bc2c96b3918cb"
Jul 16 22:20:35.366732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.366701    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:35.376440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:35.376415    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:36.494794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.494769    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:36.497704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.497682    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:36.585483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.585442    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca\") pod \"ff162e5c-31bc-4a4a-ada4-008b2f358596\" (UID: \"ff162e5c-31bc-4a4a-ada4-008b2f358596\") "
Jul 16 22:20:36.585680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.585528    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca\") pod \"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac\" (UID: \"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac\") "
Jul 16 22:20:36.585813 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.585790    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff162e5c-31bc-4a4a-ada4-008b2f358596" (UID: "ff162e5c-31bc-4a4a-ada4-008b2f358596"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:36.585882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.585859    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" (UID: "1bc21ddd-1a87-48ee-8ae1-1322dfa478ac"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:20:36.687205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.687119    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff162e5c-31bc-4a4a-ada4-008b2f358596-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:20:36.687205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:36.687149    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:20:37.361812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.361777    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"
Jul 16 22:20:37.361812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.361788    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl" event={"ID":"1bc21ddd-1a87-48ee-8ae1-1322dfa478ac","Type":"ContainerDied","Data":"9f2eeac1039319cc3ab78493a92ebb326e5da38581e235ec580e399a4b4e73a7"}
Jul 16 22:20:37.362066 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.361855    2555 scope.go:117] "RemoveContainer" containerID="cddc9d49c88f5ad72cd4c72cf298adf47c77cefda0c30648f981615c4df382d2"
Jul 16 22:20:37.366659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.366547    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8" event={"ID":"ff162e5c-31bc-4a4a-ada4-008b2f358596","Type":"ContainerDied","Data":"8784c51621adbf4595c86a176e850ac8de5c1ed12f30de470afa54a9ef925eab"}
Jul 16 22:20:37.366659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.366563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"
Jul 16 22:20:37.378624 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.378607    2555 scope.go:117] "RemoveContainer" containerID="8d069920273c6b7b4f6f4818fe1027d4bcb83377e1cccdba1e7ef2c3cdf320a4"
Jul 16 22:20:37.389420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.389397    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:37.393416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.393397    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l6fzl"]
Jul 16 22:20:37.406614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.406588    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:37.411445 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.411422    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-x6qz8"]
Jul 16 22:20:37.676135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.676051    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" path="/var/lib/kubelet/pods/1bc21ddd-1a87-48ee-8ae1-1322dfa478ac/volumes"
Jul 16 22:20:37.676502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:37.676397    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" path="/var/lib/kubelet/pods/ff162e5c-31bc-4a4a-ada4-008b2f358596/volumes"
Jul 16 22:20:48.321956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.321918    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322245    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322256    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322270    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322276    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322284    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322290    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322303    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322308    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322313    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322371    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.322656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322379    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322385    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:20:48.322656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.322392    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:20:48.325237 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.325222    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:20:48.329407 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.329379    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:20:48.329563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.329515    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:20:48.334516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.334497    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:20:48.489369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.489328    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p\" (UID: \"5dc31565-5487-4a88-81e9-3a6d9ff8bb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:20:48.590283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.590184    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p\" (UID: \"5dc31565-5487-4a88-81e9-3a6d9ff8bb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:20:48.590836 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.590818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p\" (UID: \"5dc31565-5487-4a88-81e9-3a6d9ff8bb08\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:20:48.637812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.637786    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:20:48.772694 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:48.772666    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:20:48.774514 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:20:48.774488    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc31565_5487_4a88_81e9_3a6d9ff8bb08.slice/crio-97c26876f67930ea7f4ff25321f54a051fb98d4a44a1b78ce3088a4bc919a782 WatchSource:0}: Error finding container 97c26876f67930ea7f4ff25321f54a051fb98d4a44a1b78ce3088a4bc919a782: Status 404 returned error can't find the container with id 97c26876f67930ea7f4ff25321f54a051fb98d4a44a1b78ce3088a4bc919a782
Jul 16 22:20:49.436780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:49.436753    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerID="7abbb4cf7e900575d2623a0670b0e405526c0a35f9c5f46801546f1fc0d02c2a" exitCode=6
Jul 16 22:20:49.437132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:49.436839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" event={"ID":"5dc31565-5487-4a88-81e9-3a6d9ff8bb08","Type":"ContainerDied","Data":"7abbb4cf7e900575d2623a0670b0e405526c0a35f9c5f46801546f1fc0d02c2a"}
Jul 16 22:20:49.437132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:49.436905    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" event={"ID":"5dc31565-5487-4a88-81e9-3a6d9ff8bb08","Type":"ContainerStarted","Data":"97c26876f67930ea7f4ff25321f54a051fb98d4a44a1b78ce3088a4bc919a782"}
Jul 16 22:20:49.437132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:49.436986    2555 scope.go:117] "RemoveContainer" containerID="7abbb4cf7e900575d2623a0670b0e405526c0a35f9c5f46801546f1fc0d02c2a"
Jul 16 22:20:50.444866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:50.444834    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerID="a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a" exitCode=6
Jul 16 22:20:50.445257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:50.444903    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" event={"ID":"5dc31565-5487-4a88-81e9-3a6d9ff8bb08","Type":"ContainerDied","Data":"a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a"}
Jul 16 22:20:50.445257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:50.444944    2555 scope.go:117] "RemoveContainer" containerID="7abbb4cf7e900575d2623a0670b0e405526c0a35f9c5f46801546f1fc0d02c2a"
Jul 16 22:20:50.445257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:50.445084    2555 scope.go:117] "RemoveContainer" containerID="a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a"
Jul 16 22:20:50.445442 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:50.445327    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p_odh-ai-gateway-infra(5dc31565-5487-4a88-81e9-3a6d9ff8bb08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08"
Jul 16 22:20:51.451520 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:20:51.451488    2555 scope.go:117] "RemoveContainer" containerID="a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a"
Jul 16 22:20:51.451987 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:20:51.451720    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p_odh-ai-gateway-infra(5dc31565-5487-4a88-81e9-3a6d9ff8bb08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08"
Jul 16 22:21:04.670726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:04.670688    2555 scope.go:117] "RemoveContainer" containerID="a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a"
Jul 16 22:21:05.532846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:05.532810    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerID="67e0c8cb0ca465bd01a6860fd02b940eafde6e33b44680b7a1dbd77623453a31" exitCode=6
Jul 16 22:21:05.533015 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:05.532882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" event={"ID":"5dc31565-5487-4a88-81e9-3a6d9ff8bb08","Type":"ContainerDied","Data":"67e0c8cb0ca465bd01a6860fd02b940eafde6e33b44680b7a1dbd77623453a31"}
Jul 16 22:21:05.533015 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:05.532942    2555 scope.go:117] "RemoveContainer" containerID="a3bafe1435466f92fe869b273dfbe27d51dec4e0b0f70497e61f8a9f98587a8a"
Jul 16 22:21:05.533250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:05.533233    2555 scope.go:117] "RemoveContainer" containerID="67e0c8cb0ca465bd01a6860fd02b940eafde6e33b44680b7a1dbd77623453a31"
Jul 16 22:21:05.533509 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:05.533490    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p_odh-ai-gateway-infra(5dc31565-5487-4a88-81e9-3a6d9ff8bb08)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08"
Jul 16 22:21:06.560707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:06.560672    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:21:06.697207 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:06.697185    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:21:06.752479 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:06.752438    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca\") pod \"5dc31565-5487-4a88-81e9-3a6d9ff8bb08\" (UID: \"5dc31565-5487-4a88-81e9-3a6d9ff8bb08\") "
Jul 16 22:21:06.752814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:06.752791    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dc31565-5487-4a88-81e9-3a6d9ff8bb08" (UID: "5dc31565-5487-4a88-81e9-3a6d9ff8bb08"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:06.853901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:06.853816    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dc31565-5487-4a88-81e9-3a6d9ff8bb08-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:21:07.550806 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.550776    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"
Jul 16 22:21:07.550988 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.550818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p" event={"ID":"5dc31565-5487-4a88-81e9-3a6d9ff8bb08","Type":"ContainerDied","Data":"97c26876f67930ea7f4ff25321f54a051fb98d4a44a1b78ce3088a4bc919a782"}
Jul 16 22:21:07.550988 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.550919    2555 scope.go:117] "RemoveContainer" containerID="67e0c8cb0ca465bd01a6860fd02b940eafde6e33b44680b7a1dbd77623453a31"
Jul 16 22:21:07.573869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.573841    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:21:07.581473 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.581438    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gkq8p"]
Jul 16 22:21:07.675573 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:07.675537    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" path="/var/lib/kubelet/pods/5dc31565-5487-4a88-81e9-3a6d9ff8bb08/volumes"
Jul 16 22:21:09.605350 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605312    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:09.605856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605814    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.605856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605833    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.605973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605863    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.605973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.605973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605893    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:21:09.605973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605903    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:21:09.606178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.605990    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff162e5c-31bc-4a4a-ada4-008b2f358596" containerName="revoke-keys"
Jul 16 22:21:09.606178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.606004    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1bc21ddd-1a87-48ee-8ae1-1322dfa478ac" containerName="revoke-keys"
Jul 16 22:21:09.606178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.606013    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.606178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.606022    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.606178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.606031    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:09.610194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.610172    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:09.612792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.612772    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:09.612981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.612968    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:21:09.620603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.620584    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:09.678366 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.678329    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl\" (UID: \"6a9447ae-60f6-4299-ab3e-aac26496e9cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:09.778831 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.778795    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl\" (UID: \"6a9447ae-60f6-4299-ab3e-aac26496e9cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:09.779423 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.779403    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl\" (UID: \"6a9447ae-60f6-4299-ab3e-aac26496e9cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:09.921205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:09.921113    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:10.057630 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.057603    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:10.059086 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:21:10.059058    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a9447ae_60f6_4299_ab3e_aac26496e9cd.slice/crio-96ad763329d975ceaef9b205389d3e8466614f81630522379dbdf0cf5d094372 WatchSource:0}: Error finding container 96ad763329d975ceaef9b205389d3e8466614f81630522379dbdf0cf5d094372: Status 404 returned error can't find the container with id 96ad763329d975ceaef9b205389d3e8466614f81630522379dbdf0cf5d094372
Jul 16 22:21:10.122608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.122578    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:10.122929 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.122917    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:10.122972 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.122931    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc31565-5487-4a88-81e9-3a6d9ff8bb08" containerName="revoke-keys"
Jul 16 22:21:10.126895 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.126880    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:10.135797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.135754    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:10.181017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.180919    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tz78w\" (UID: \"cd3fd156-983c-49d7-8a9d-edc3fea8ef49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:10.282285 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.282251    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tz78w\" (UID: \"cd3fd156-983c-49d7-8a9d-edc3fea8ef49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:10.282893 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.282875    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tz78w\" (UID: \"cd3fd156-983c-49d7-8a9d-edc3fea8ef49\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:10.439284 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.439199    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:10.568153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.568120    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerStarted","Data":"7f874a0ba495540f938b81a9392e64b4f73ad9644288771c430b936e7f62ffbd"}
Jul 16 22:21:10.568321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.568180    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerStarted","Data":"96ad763329d975ceaef9b205389d3e8466614f81630522379dbdf0cf5d094372"}
Jul 16 22:21:10.590999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.590978    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:10.592395 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:21:10.592373    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd3fd156_983c_49d7_8a9d_edc3fea8ef49.slice/crio-7c1156202caf550af4c39e10c0313e7b36455ea420d5edb36fbfba68872d90a1 WatchSource:0}: Error finding container 7c1156202caf550af4c39e10c0313e7b36455ea420d5edb36fbfba68872d90a1: Status 404 returned error can't find the container with id 7c1156202caf550af4c39e10c0313e7b36455ea420d5edb36fbfba68872d90a1
Jul 16 22:21:10.597272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:10.597230    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" podStartSLOduration=1.597216915 podStartE2EDuration="1.597216915s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:21:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:21:10.595545303 +0000 UTC m=+1925.432599103" watchObservedRunningTime="2026-07-16 22:21:10.597216915 +0000 UTC m=+1925.434270770"
Jul 16 22:21:11.575068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.575031    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerID="7f874a0ba495540f938b81a9392e64b4f73ad9644288771c430b936e7f62ffbd" exitCode=6
Jul 16 22:21:11.575574 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.575118    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerDied","Data":"7f874a0ba495540f938b81a9392e64b4f73ad9644288771c430b936e7f62ffbd"}
Jul 16 22:21:11.575574 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.575332    2555 scope.go:117] "RemoveContainer" containerID="7f874a0ba495540f938b81a9392e64b4f73ad9644288771c430b936e7f62ffbd"
Jul 16 22:21:11.576822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.576798    2555 generic.go:356] "Generic (PLEG): container finished" podID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerID="9dfb07747da9544ce3e2311ffad10fdc9a1d20fa06d10c815d627e444fd9a08f" exitCode=7
Jul 16 22:21:11.576946 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.576889    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" event={"ID":"cd3fd156-983c-49d7-8a9d-edc3fea8ef49","Type":"ContainerDied","Data":"9dfb07747da9544ce3e2311ffad10fdc9a1d20fa06d10c815d627e444fd9a08f"}
Jul 16 22:21:11.576946 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.576924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" event={"ID":"cd3fd156-983c-49d7-8a9d-edc3fea8ef49","Type":"ContainerStarted","Data":"7c1156202caf550af4c39e10c0313e7b36455ea420d5edb36fbfba68872d90a1"}
Jul 16 22:21:11.577071 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:11.577032    2555 scope.go:117] "RemoveContainer" containerID="9dfb07747da9544ce3e2311ffad10fdc9a1d20fa06d10c815d627e444fd9a08f"
Jul 16 22:21:12.583824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.583742    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerID="db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58" exitCode=6
Jul 16 22:21:12.584329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.583829    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerDied","Data":"db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58"}
Jul 16 22:21:12.584329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.583935    2555 scope.go:117] "RemoveContainer" containerID="7f874a0ba495540f938b81a9392e64b4f73ad9644288771c430b936e7f62ffbd"
Jul 16 22:21:12.584329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.584200    2555 scope.go:117] "RemoveContainer" containerID="db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58"
Jul 16 22:21:12.584511 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:12.584493    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl_odh-ai-gateway-infra(6a9447ae-60f6-4299-ab3e-aac26496e9cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd"
Jul 16 22:21:12.585807 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.585784    2555 generic.go:356] "Generic (PLEG): container finished" podID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerID="5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976" exitCode=7
Jul 16 22:21:12.585921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.585866    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" event={"ID":"cd3fd156-983c-49d7-8a9d-edc3fea8ef49","Type":"ContainerDied","Data":"5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976"}
Jul 16 22:21:12.586051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.586036    2555 scope.go:117] "RemoveContainer" containerID="5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976"
Jul 16 22:21:12.586218 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:12.586202    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tz78w_odh-ai-gateway-infra(cd3fd156-983c-49d7-8a9d-edc3fea8ef49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49"
Jul 16 22:21:12.595892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:12.595872    2555 scope.go:117] "RemoveContainer" containerID="9dfb07747da9544ce3e2311ffad10fdc9a1d20fa06d10c815d627e444fd9a08f"
Jul 16 22:21:13.592809 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:13.592782    2555 scope.go:117] "RemoveContainer" containerID="db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58"
Jul 16 22:21:13.593215 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:13.593017    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl_odh-ai-gateway-infra(6a9447ae-60f6-4299-ab3e-aac26496e9cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd"
Jul 16 22:21:13.594287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:13.594273    2555 scope.go:117] "RemoveContainer" containerID="5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976"
Jul 16 22:21:13.594447 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:13.594431    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tz78w_odh-ai-gateway-infra(cd3fd156-983c-49d7-8a9d-edc3fea8ef49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49"
Jul 16 22:21:24.670354 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:24.670326    2555 scope.go:117] "RemoveContainer" containerID="5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976"
Jul 16 22:21:25.659508 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:25.659396    2555 generic.go:356] "Generic (PLEG): container finished" podID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerID="63dd252a660cd075e0991f44cd4895ab33807e7ecbdc7aee639e72a787bfb485" exitCode=7
Jul 16 22:21:25.659508 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:25.659481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" event={"ID":"cd3fd156-983c-49d7-8a9d-edc3fea8ef49","Type":"ContainerDied","Data":"63dd252a660cd075e0991f44cd4895ab33807e7ecbdc7aee639e72a787bfb485"}
Jul 16 22:21:25.659746 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:25.659543    2555 scope.go:117] "RemoveContainer" containerID="5278b56aac4a07fda2dba55e1d56540b02b02d391bd6686cfa70a9600eebd976"
Jul 16 22:21:25.659839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:25.659824    2555 scope.go:117] "RemoveContainer" containerID="63dd252a660cd075e0991f44cd4895ab33807e7ecbdc7aee639e72a787bfb485"
Jul 16 22:21:25.660068 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:25.660040    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tz78w_odh-ai-gateway-infra(cd3fd156-983c-49d7-8a9d-edc3fea8ef49)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49"
Jul 16 22:21:26.670825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:26.670796    2555 scope.go:117] "RemoveContainer" containerID="db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58"
Jul 16 22:21:26.688756 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:26.688732    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:26.841209 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:26.841186    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:26.931073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:26.931014    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca\") pod \"cd3fd156-983c-49d7-8a9d-edc3fea8ef49\" (UID: \"cd3fd156-983c-49d7-8a9d-edc3fea8ef49\") "
Jul 16 22:21:26.931314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:26.931294    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd3fd156-983c-49d7-8a9d-edc3fea8ef49" (UID: "cd3fd156-983c-49d7-8a9d-edc3fea8ef49"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:27.032415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.032383    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd3fd156-983c-49d7-8a9d-edc3fea8ef49-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:21:27.675722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.675631    2555 generic.go:356] "Generic (PLEG): container finished" podID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerID="09a466e81f4b84d3309e8d54dc3bca4be173473a554c6e8b7fde4e757b2b3a38" exitCode=6
Jul 16 22:21:27.677033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.677012    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"
Jul 16 22:21:27.679952 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.679910    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerDied","Data":"09a466e81f4b84d3309e8d54dc3bca4be173473a554c6e8b7fde4e757b2b3a38"}
Jul 16 22:21:27.680071 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.679976    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w" event={"ID":"cd3fd156-983c-49d7-8a9d-edc3fea8ef49","Type":"ContainerDied","Data":"7c1156202caf550af4c39e10c0313e7b36455ea420d5edb36fbfba68872d90a1"}
Jul 16 22:21:27.680071 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.680012    2555 scope.go:117] "RemoveContainer" containerID="db68cf2f21b167edd56bfa63e42175edf12054695c474611e28df97b053c0a58"
Jul 16 22:21:27.680359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.680329    2555 scope.go:117] "RemoveContainer" containerID="09a466e81f4b84d3309e8d54dc3bca4be173473a554c6e8b7fde4e757b2b3a38"
Jul 16 22:21:27.680629 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:27.680603    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl_odh-ai-gateway-infra(6a9447ae-60f6-4299-ab3e-aac26496e9cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd"
Jul 16 22:21:27.692865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.692848    2555 scope.go:117] "RemoveContainer" containerID="63dd252a660cd075e0991f44cd4895ab33807e7ecbdc7aee639e72a787bfb485"
Jul 16 22:21:27.710692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.710666    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:27.716745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:27.716724    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tz78w"]
Jul 16 22:21:28.705016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:28.704977    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:28.838608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:28.838583    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:28.948871 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:28.948838    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca\") pod \"6a9447ae-60f6-4299-ab3e-aac26496e9cd\" (UID: \"6a9447ae-60f6-4299-ab3e-aac26496e9cd\") "
Jul 16 22:21:28.949226 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:28.949204    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6a9447ae-60f6-4299-ab3e-aac26496e9cd" (UID: "6a9447ae-60f6-4299-ab3e-aac26496e9cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:29.049530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.049498    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6a9447ae-60f6-4299-ab3e-aac26496e9cd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:21:29.676056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.676023    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" path="/var/lib/kubelet/pods/cd3fd156-983c-49d7-8a9d-edc3fea8ef49/volumes"
Jul 16 22:21:29.690674 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.690643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl" event={"ID":"6a9447ae-60f6-4299-ab3e-aac26496e9cd","Type":"ContainerDied","Data":"96ad763329d975ceaef9b205389d3e8466614f81630522379dbdf0cf5d094372"}
Jul 16 22:21:29.690859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.690701    2555 scope.go:117] "RemoveContainer" containerID="09a466e81f4b84d3309e8d54dc3bca4be173473a554c6e8b7fde4e757b2b3a38"
Jul 16 22:21:29.690859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.690656    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"
Jul 16 22:21:29.708204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.708167    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:29.713776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:29.713748    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7b6tl"]
Jul 16 22:21:31.676797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:31.676763    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" path="/var/lib/kubelet/pods/6a9447ae-60f6-4299-ab3e-aac26496e9cd/volumes"
Jul 16 22:21:38.375555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375527    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375852    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375863    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375877    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375883    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375893    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375898    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375907    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375911    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375918    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.375919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375923    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.376239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375978    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.376239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375986    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.376239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.375993    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.376239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.376000    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:21:38.376239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.376005    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:21:38.378932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.378913    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:38.381539 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.381514    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:21:38.382614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.382576    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:21:38.388004 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.387977    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:38.532953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.532914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz\" (UID: \"f29e85bc-2cb0-4198-ad57-9e40cc1c7408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:38.634328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.634249    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz\" (UID: \"f29e85bc-2cb0-4198-ad57-9e40cc1c7408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:38.634898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.634876    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz\" (UID: \"f29e85bc-2cb0-4198-ad57-9e40cc1c7408\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:38.690568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.690537    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:38.833946 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:38.833920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:38.836263 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:21:38.836235    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf29e85bc_2cb0_4198_ad57_9e40cc1c7408.slice/crio-5d34978ea2bed053542c2f93939fc96304174cf2cce478fe6dd69eb6ec6cdbc3 WatchSource:0}: Error finding container 5d34978ea2bed053542c2f93939fc96304174cf2cce478fe6dd69eb6ec6cdbc3: Status 404 returned error can't find the container with id 5d34978ea2bed053542c2f93939fc96304174cf2cce478fe6dd69eb6ec6cdbc3
Jul 16 22:21:39.742606 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:39.742509    2555 generic.go:356] "Generic (PLEG): container finished" podID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerID="57114ca01aed32da2e953e6718287ee946ace435ac13fd7a48241caf93ef3069" exitCode=6
Jul 16 22:21:39.742953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:39.742601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" event={"ID":"f29e85bc-2cb0-4198-ad57-9e40cc1c7408","Type":"ContainerDied","Data":"57114ca01aed32da2e953e6718287ee946ace435ac13fd7a48241caf93ef3069"}
Jul 16 22:21:39.742953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:39.742658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" event={"ID":"f29e85bc-2cb0-4198-ad57-9e40cc1c7408","Type":"ContainerStarted","Data":"5d34978ea2bed053542c2f93939fc96304174cf2cce478fe6dd69eb6ec6cdbc3"}
Jul 16 22:21:39.742953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:39.742773    2555 scope.go:117] "RemoveContainer" containerID="57114ca01aed32da2e953e6718287ee946ace435ac13fd7a48241caf93ef3069"
Jul 16 22:21:40.750033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:40.749944    2555 generic.go:356] "Generic (PLEG): container finished" podID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerID="209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4" exitCode=6
Jul 16 22:21:40.750449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:40.750033    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" event={"ID":"f29e85bc-2cb0-4198-ad57-9e40cc1c7408","Type":"ContainerDied","Data":"209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4"}
Jul 16 22:21:40.750449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:40.750100    2555 scope.go:117] "RemoveContainer" containerID="57114ca01aed32da2e953e6718287ee946ace435ac13fd7a48241caf93ef3069"
Jul 16 22:21:40.750449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:40.750236    2555 scope.go:117] "RemoveContainer" containerID="209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4"
Jul 16 22:21:40.750582 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:40.750443    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz_odh-ai-gateway-infra(f29e85bc-2cb0-4198-ad57-9e40cc1c7408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408"
Jul 16 22:21:41.757154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:41.757123    2555 scope.go:117] "RemoveContainer" containerID="209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4"
Jul 16 22:21:41.757571 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:41.757307    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz_odh-ai-gateway-infra(f29e85bc-2cb0-4198-ad57-9e40cc1c7408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408"
Jul 16 22:21:52.670846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:52.670813    2555 scope.go:117] "RemoveContainer" containerID="209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4"
Jul 16 22:21:53.824720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:53.824682    2555 generic.go:356] "Generic (PLEG): container finished" podID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerID="71dd5e5072f36d3e8eef61ba79da7a65fd5df91630dd45336c53af35265ed613" exitCode=6
Jul 16 22:21:53.825183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:53.824765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" event={"ID":"f29e85bc-2cb0-4198-ad57-9e40cc1c7408","Type":"ContainerDied","Data":"71dd5e5072f36d3e8eef61ba79da7a65fd5df91630dd45336c53af35265ed613"}
Jul 16 22:21:53.825183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:53.824831    2555 scope.go:117] "RemoveContainer" containerID="209d0534b8c48c2cf11792b1ec1816b897b13280719c49692f6dc07811765bd4"
Jul 16 22:21:53.825183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:53.825160    2555 scope.go:117] "RemoveContainer" containerID="71dd5e5072f36d3e8eef61ba79da7a65fd5df91630dd45336c53af35265ed613"
Jul 16 22:21:53.825427 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:21:53.825407    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz_odh-ai-gateway-infra(f29e85bc-2cb0-4198-ad57-9e40cc1c7408)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408"
Jul 16 22:21:54.857778 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:54.857747    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:54.996411 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:54.996388    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:55.080375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.080345    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca\") pod \"f29e85bc-2cb0-4198-ad57-9e40cc1c7408\" (UID: \"f29e85bc-2cb0-4198-ad57-9e40cc1c7408\") "
Jul 16 22:21:55.080756 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.080731    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f29e85bc-2cb0-4198-ad57-9e40cc1c7408" (UID: "f29e85bc-2cb0-4198-ad57-9e40cc1c7408"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:21:55.181643 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.181538    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f29e85bc-2cb0-4198-ad57-9e40cc1c7408-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:21:55.838091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.838016    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"
Jul 16 22:21:55.838091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.838065    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz" event={"ID":"f29e85bc-2cb0-4198-ad57-9e40cc1c7408","Type":"ContainerDied","Data":"5d34978ea2bed053542c2f93939fc96304174cf2cce478fe6dd69eb6ec6cdbc3"}
Jul 16 22:21:55.838380 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.838120    2555 scope.go:117] "RemoveContainer" containerID="71dd5e5072f36d3e8eef61ba79da7a65fd5df91630dd45336c53af35265ed613"
Jul 16 22:21:55.861541 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.861516    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:55.868181 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:55.868159    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-zj2pz"]
Jul 16 22:21:57.675814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:21:57.675779    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" path="/var/lib/kubelet/pods/f29e85bc-2cb0-4198-ad57-9e40cc1c7408/volumes"
Jul 16 22:22:00.185631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185594    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185944    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185955    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185962    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185979    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185984    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.185996    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:22:00.186010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.186001    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a9447ae-60f6-4299-ab3e-aac26496e9cd" containerName="revoke-keys"
Jul 16 22:22:00.186269 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.186059    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186269 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.186066    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:00.186269 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.186073    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd3fd156-983c-49d7-8a9d-edc3fea8ef49" containerName="revoke-keys"
Jul 16 22:22:00.190627 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.190609    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:00.199872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.199836    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:22:00.200131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.200112    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:00.201780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.201754    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:00.322588 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.322557    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs2vr\" (UID: \"36ddc604-960a-4337-9e96-c3c9f069a1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:00.423491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.423428    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs2vr\" (UID: \"36ddc604-960a-4337-9e96-c3c9f069a1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:00.424056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.424036    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs2vr\" (UID: \"36ddc604-960a-4337-9e96-c3c9f069a1ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:00.501948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.501850    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:00.649775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.649747    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:00.651731 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:22:00.651700    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36ddc604_960a_4337_9e96_c3c9f069a1ae.slice/crio-e5c5d36d52efc15c1ddcbb0824f8693bd5d28c2a53d22230eeb6e5a6535d4c0e WatchSource:0}: Error finding container e5c5d36d52efc15c1ddcbb0824f8693bd5d28c2a53d22230eeb6e5a6535d4c0e: Status 404 returned error can't find the container with id e5c5d36d52efc15c1ddcbb0824f8693bd5d28c2a53d22230eeb6e5a6535d4c0e
Jul 16 22:22:00.872546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:00.872507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" event={"ID":"36ddc604-960a-4337-9e96-c3c9f069a1ae","Type":"ContainerStarted","Data":"e5c5d36d52efc15c1ddcbb0824f8693bd5d28c2a53d22230eeb6e5a6535d4c0e"}
Jul 16 22:22:01.879000 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:01.878957    2555 generic.go:356] "Generic (PLEG): container finished" podID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerID="c63e0c2a981af6d5d71b3568534cf01645ee794fd8dd3441e14df01b156696bc" exitCode=7
Jul 16 22:22:01.879000 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:01.879005    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" event={"ID":"36ddc604-960a-4337-9e96-c3c9f069a1ae","Type":"ContainerDied","Data":"c63e0c2a981af6d5d71b3568534cf01645ee794fd8dd3441e14df01b156696bc"}
Jul 16 22:22:01.879477 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:01.879254    2555 scope.go:117] "RemoveContainer" containerID="c63e0c2a981af6d5d71b3568534cf01645ee794fd8dd3441e14df01b156696bc"
Jul 16 22:22:02.886455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:02.886424    2555 generic.go:356] "Generic (PLEG): container finished" podID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerID="54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e" exitCode=7
Jul 16 22:22:02.886948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:02.886507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" event={"ID":"36ddc604-960a-4337-9e96-c3c9f069a1ae","Type":"ContainerDied","Data":"54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e"}
Jul 16 22:22:02.886948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:02.886566    2555 scope.go:117] "RemoveContainer" containerID="c63e0c2a981af6d5d71b3568534cf01645ee794fd8dd3441e14df01b156696bc"
Jul 16 22:22:02.886948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:02.886708    2555 scope.go:117] "RemoveContainer" containerID="54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e"
Jul 16 22:22:02.887118 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:02.886947    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs2vr_odh-ai-gateway-infra(36ddc604-960a-4337-9e96-c3c9f069a1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae"
Jul 16 22:22:03.894364 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:03.894333    2555 scope.go:117] "RemoveContainer" containerID="54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e"
Jul 16 22:22:03.894805 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:03.894531    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs2vr_odh-ai-gateway-infra(36ddc604-960a-4337-9e96-c3c9f069a1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae"
Jul 16 22:22:09.680240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.680209    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:09.680673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.680658    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f29e85bc-2cb0-4198-ad57-9e40cc1c7408" containerName="revoke-keys"
Jul 16 22:22:09.683561 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.683543    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:09.696983 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.696958    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:09.808567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.808522    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg\" (UID: \"d5abc1fe-b5bb-4170-b3c3-796382269172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:09.909894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.909851    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg\" (UID: \"d5abc1fe-b5bb-4170-b3c3-796382269172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:09.910571 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.910545    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg\" (UID: \"d5abc1fe-b5bb-4170-b3c3-796382269172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:09.992892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:09.992795    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:10.128719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:10.128689    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:10.132547 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:22:10.132521    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5abc1fe_b5bb_4170_b3c3_796382269172.slice/crio-89b2b68269eb774410e994ec4c8b4d94ee5a6ca4b30211d0d8c7f6d66e3c5f75 WatchSource:0}: Error finding container 89b2b68269eb774410e994ec4c8b4d94ee5a6ca4b30211d0d8c7f6d66e3c5f75: Status 404 returned error can't find the container with id 89b2b68269eb774410e994ec4c8b4d94ee5a6ca4b30211d0d8c7f6d66e3c5f75
Jul 16 22:22:10.933918 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:10.933884    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerID="cf537b8b6296a3790ab71d301c453efe5e1356b6190f12a046ed8ff2e35e98c7" exitCode=6
Jul 16 22:22:10.934355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:10.933936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" event={"ID":"d5abc1fe-b5bb-4170-b3c3-796382269172","Type":"ContainerDied","Data":"cf537b8b6296a3790ab71d301c453efe5e1356b6190f12a046ed8ff2e35e98c7"}
Jul 16 22:22:10.934355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:10.933987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" event={"ID":"d5abc1fe-b5bb-4170-b3c3-796382269172","Type":"ContainerStarted","Data":"89b2b68269eb774410e994ec4c8b4d94ee5a6ca4b30211d0d8c7f6d66e3c5f75"}
Jul 16 22:22:10.934355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:10.934227    2555 scope.go:117] "RemoveContainer" containerID="cf537b8b6296a3790ab71d301c453efe5e1356b6190f12a046ed8ff2e35e98c7"
Jul 16 22:22:11.940835 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:11.940798    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerID="f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5" exitCode=6
Jul 16 22:22:11.941288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:11.940875    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" event={"ID":"d5abc1fe-b5bb-4170-b3c3-796382269172","Type":"ContainerDied","Data":"f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5"}
Jul 16 22:22:11.941288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:11.940928    2555 scope.go:117] "RemoveContainer" containerID="cf537b8b6296a3790ab71d301c453efe5e1356b6190f12a046ed8ff2e35e98c7"
Jul 16 22:22:11.941288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:11.941187    2555 scope.go:117] "RemoveContainer" containerID="f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5"
Jul 16 22:22:11.941428 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:11.941387    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg_odh-ai-gateway-infra(d5abc1fe-b5bb-4170-b3c3-796382269172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172"
Jul 16 22:22:12.948341 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:12.948310    2555 scope.go:117] "RemoveContainer" containerID="f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5"
Jul 16 22:22:12.948963 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:12.948648    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg_odh-ai-gateway-infra(d5abc1fe-b5bb-4170-b3c3-796382269172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172"
Jul 16 22:22:15.672959 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:15.672936    2555 scope.go:117] "RemoveContainer" containerID="54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e"
Jul 16 22:22:16.972551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:16.972518    2555 generic.go:356] "Generic (PLEG): container finished" podID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerID="b5e6348abc9af75db8cddc06c43c45a2b4866693215768c14e820ec1a6acca25" exitCode=7
Jul 16 22:22:16.973135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:16.972574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" event={"ID":"36ddc604-960a-4337-9e96-c3c9f069a1ae","Type":"ContainerDied","Data":"b5e6348abc9af75db8cddc06c43c45a2b4866693215768c14e820ec1a6acca25"}
Jul 16 22:22:16.973135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:16.972623    2555 scope.go:117] "RemoveContainer" containerID="54653ceeb9cbd6a8cdc47ad649ffbdf87f73c8a3e706620cd368741b0a517c2e"
Jul 16 22:22:16.973135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:16.972942    2555 scope.go:117] "RemoveContainer" containerID="b5e6348abc9af75db8cddc06c43c45a2b4866693215768c14e820ec1a6acca25"
Jul 16 22:22:16.973320 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:16.973200    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs2vr_odh-ai-gateway-infra(36ddc604-960a-4337-9e96-c3c9f069a1ae)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae"
Jul 16 22:22:18.005339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.005303    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:18.145154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.145131    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:18.281447 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.281420    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca\") pod \"36ddc604-960a-4337-9e96-c3c9f069a1ae\" (UID: \"36ddc604-960a-4337-9e96-c3c9f069a1ae\") "
Jul 16 22:22:18.281858 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.281837    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "36ddc604-960a-4337-9e96-c3c9f069a1ae" (UID: "36ddc604-960a-4337-9e96-c3c9f069a1ae"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:18.382651 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.382616    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/36ddc604-960a-4337-9e96-c3c9f069a1ae-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:22:18.986690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.986656    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"
Jul 16 22:22:18.986894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.986691    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr" event={"ID":"36ddc604-960a-4337-9e96-c3c9f069a1ae","Type":"ContainerDied","Data":"e5c5d36d52efc15c1ddcbb0824f8693bd5d28c2a53d22230eeb6e5a6535d4c0e"}
Jul 16 22:22:18.986894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:18.986770    2555 scope.go:117] "RemoveContainer" containerID="b5e6348abc9af75db8cddc06c43c45a2b4866693215768c14e820ec1a6acca25"
Jul 16 22:22:19.009738 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:19.009708    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:19.013017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:19.012993    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs2vr"]
Jul 16 22:22:19.676261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:19.676220    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" path="/var/lib/kubelet/pods/36ddc604-960a-4337-9e96-c3c9f069a1ae/volumes"
Jul 16 22:22:23.670958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:23.670929    2555 scope.go:117] "RemoveContainer" containerID="f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5"
Jul 16 22:22:25.021631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:25.021595    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerID="a89232437dab152d6790ecea7b2b977ffaaba79f18fb0f91dcced10a626b7d86" exitCode=6
Jul 16 22:22:25.022086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:25.021664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" event={"ID":"d5abc1fe-b5bb-4170-b3c3-796382269172","Type":"ContainerDied","Data":"a89232437dab152d6790ecea7b2b977ffaaba79f18fb0f91dcced10a626b7d86"}
Jul 16 22:22:25.022086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:25.021725    2555 scope.go:117] "RemoveContainer" containerID="f7398b3ff8c1d8520654c444d794569400746047be679c06614cb2f50136b6d5"
Jul 16 22:22:25.022086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:25.022022    2555 scope.go:117] "RemoveContainer" containerID="a89232437dab152d6790ecea7b2b977ffaaba79f18fb0f91dcced10a626b7d86"
Jul 16 22:22:25.022257 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:25.022222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg_odh-ai-gateway-infra(d5abc1fe-b5bb-4170-b3c3-796382269172)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172"
Jul 16 22:22:26.051262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:26.051231    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:26.185998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:26.185977    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:26.353406 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:26.353307    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca\") pod \"d5abc1fe-b5bb-4170-b3c3-796382269172\" (UID: \"d5abc1fe-b5bb-4170-b3c3-796382269172\") "
Jul 16 22:22:26.353695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:26.353672    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5abc1fe-b5bb-4170-b3c3-796382269172" (UID: "d5abc1fe-b5bb-4170-b3c3-796382269172"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:26.454776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:26.454739    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5abc1fe-b5bb-4170-b3c3-796382269172-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:22:27.034901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.034873    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"
Jul 16 22:22:27.034901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.034886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg" event={"ID":"d5abc1fe-b5bb-4170-b3c3-796382269172","Type":"ContainerDied","Data":"89b2b68269eb774410e994ec4c8b4d94ee5a6ca4b30211d0d8c7f6d66e3c5f75"}
Jul 16 22:22:27.035134 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.034949    2555 scope.go:117] "RemoveContainer" containerID="a89232437dab152d6790ecea7b2b977ffaaba79f18fb0f91dcced10a626b7d86"
Jul 16 22:22:27.056253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.056227    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:27.064246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.064219    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lm4fg"]
Jul 16 22:22:27.676377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:27.676335    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" path="/var/lib/kubelet/pods/d5abc1fe-b5bb-4170-b3c3-796382269172/volumes"
Jul 16 22:22:28.455345 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455313    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455660    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455672    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455685    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455691    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455699    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455705    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455712    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455717    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455733    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.455744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455738    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.456107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455784    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.456107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455792    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.456107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455798    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.456107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455802    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5abc1fe-b5bb-4170-b3c3-796382269172" containerName="revoke-keys"
Jul 16 22:22:28.456107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.455809    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:28.460286 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.460270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:28.462737 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.462715    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:28.462866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.462778    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:22:28.467333 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.467314    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:28.576447 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.576413    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9\" (UID: \"93621d81-8c2f-4adc-96b0-8e468bd25a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:28.677753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.677721    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9\" (UID: \"93621d81-8c2f-4adc-96b0-8e468bd25a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:28.678361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.678342    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9\" (UID: \"93621d81-8c2f-4adc-96b0-8e468bd25a30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:28.773051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.772952    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:28.905574 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:28.905548    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:28.907537 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:22:28.907511    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93621d81_8c2f_4adc_96b0_8e468bd25a30.slice/crio-a8aa7a05ead130e21f5872ee225089e32958b7906c5f858f54cb5e3d039bd9ef WatchSource:0}: Error finding container a8aa7a05ead130e21f5872ee225089e32958b7906c5f858f54cb5e3d039bd9ef: Status 404 returned error can't find the container with id a8aa7a05ead130e21f5872ee225089e32958b7906c5f858f54cb5e3d039bd9ef
Jul 16 22:22:29.047053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:29.047017    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerStarted","Data":"a8aa7a05ead130e21f5872ee225089e32958b7906c5f858f54cb5e3d039bd9ef"}
Jul 16 22:22:30.054253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:30.054216    2555 generic.go:356] "Generic (PLEG): container finished" podID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerID="981641296de399826efe64264fc797dd88af3a5f3395d5b31197914a1e21756b" exitCode=6
Jul 16 22:22:30.054658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:30.054285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerDied","Data":"981641296de399826efe64264fc797dd88af3a5f3395d5b31197914a1e21756b"}
Jul 16 22:22:30.054658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:30.054452    2555 scope.go:117] "RemoveContainer" containerID="981641296de399826efe64264fc797dd88af3a5f3395d5b31197914a1e21756b"
Jul 16 22:22:31.061732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:31.061701    2555 generic.go:356] "Generic (PLEG): container finished" podID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerID="b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4" exitCode=6
Jul 16 22:22:31.062190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:31.061752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerDied","Data":"b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4"}
Jul 16 22:22:31.062190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:31.061800    2555 scope.go:117] "RemoveContainer" containerID="981641296de399826efe64264fc797dd88af3a5f3395d5b31197914a1e21756b"
Jul 16 22:22:31.062190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:31.062007    2555 scope.go:117] "RemoveContainer" containerID="b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4"
Jul 16 22:22:31.062345 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:31.062206    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9_odh-ai-gateway-infra(93621d81-8c2f-4adc-96b0-8e468bd25a30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30"
Jul 16 22:22:32.068245 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:32.068216    2555 scope.go:117] "RemoveContainer" containerID="b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4"
Jul 16 22:22:32.068643 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:32.068425    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9_odh-ai-gateway-infra(93621d81-8c2f-4adc-96b0-8e468bd25a30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30"
Jul 16 22:22:43.671277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:43.671243    2555 scope.go:117] "RemoveContainer" containerID="b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4"
Jul 16 22:22:44.136237 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:44.136199    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerStarted","Data":"f83396d9d598f2395ff14adc0c780fec8e3faee86b2606451ca14c19da9993f5"}
Jul 16 22:22:44.154711 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:44.154658    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" podStartSLOduration=16.15464379 podStartE2EDuration="16.15464379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:44.152855981 +0000 UTC m=+2018.989909803" watchObservedRunningTime="2026-07-16 22:22:44.15464379 +0000 UTC m=+2018.991697590"
Jul 16 22:22:45.144196 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:45.144156    2555 generic.go:356] "Generic (PLEG): container finished" podID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerID="f83396d9d598f2395ff14adc0c780fec8e3faee86b2606451ca14c19da9993f5" exitCode=6
Jul 16 22:22:45.144646 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:45.144235    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerDied","Data":"f83396d9d598f2395ff14adc0c780fec8e3faee86b2606451ca14c19da9993f5"}
Jul 16 22:22:45.144646 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:45.144296    2555 scope.go:117] "RemoveContainer" containerID="b33716304fa5d2e0b46750a8b06129fba34a0190329745f8d44bc9e64e0eede4"
Jul 16 22:22:45.144763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:45.144663    2555 scope.go:117] "RemoveContainer" containerID="f83396d9d598f2395ff14adc0c780fec8e3faee86b2606451ca14c19da9993f5"
Jul 16 22:22:45.144886 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:45.144866    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9_odh-ai-gateway-infra(93621d81-8c2f-4adc-96b0-8e468bd25a30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30"
Jul 16 22:22:45.172240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:45.172211    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:46.291871 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:46.291844    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:46.329602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:46.329576    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca\") pod \"93621d81-8c2f-4adc-96b0-8e468bd25a30\" (UID: \"93621d81-8c2f-4adc-96b0-8e468bd25a30\") "
Jul 16 22:22:46.329966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:46.329947    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "93621d81-8c2f-4adc-96b0-8e468bd25a30" (UID: "93621d81-8c2f-4adc-96b0-8e468bd25a30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:46.430283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:46.430198    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/93621d81-8c2f-4adc-96b0-8e468bd25a30-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:22:47.157098 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.157067    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"
Jul 16 22:22:47.157298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.157112    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9" event={"ID":"93621d81-8c2f-4adc-96b0-8e468bd25a30","Type":"ContainerDied","Data":"a8aa7a05ead130e21f5872ee225089e32958b7906c5f858f54cb5e3d039bd9ef"}
Jul 16 22:22:47.157298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.157175    2555 scope.go:117] "RemoveContainer" containerID="f83396d9d598f2395ff14adc0c780fec8e3faee86b2606451ca14c19da9993f5"
Jul 16 22:22:47.180741 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.180711    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:47.189116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.189090    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-rfqj9"]
Jul 16 22:22:47.675889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:47.675858    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" path="/var/lib/kubelet/pods/93621d81-8c2f-4adc-96b0-8e468bd25a30/volumes"
Jul 16 22:22:50.245896 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.245861    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246206    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246216    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246226    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246231    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246246    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246251    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246493 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246298    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246493 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246305    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:50.246493 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.246312    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="36ddc604-960a-4337-9e96-c3c9f069a1ae" containerName="revoke-keys"
Jul 16 22:22:50.250850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.250832    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:22:50.255930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.255901    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:22:50.256057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.255902    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:22:50.262230 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.262208    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:22:50.364374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.364333    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-mq6vh\" (UID: \"2e668468-6933-4cd2-9050-180c93627d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:22:50.464782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.464749    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-mq6vh\" (UID: \"2e668468-6933-4cd2-9050-180c93627d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:22:50.465339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.465318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-mq6vh\" (UID: \"2e668468-6933-4cd2-9050-180c93627d7b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:22:50.561183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.561155    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:22:50.706090 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:50.706055    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:22:50.707559 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:22:50.707518    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e668468_6933_4cd2_9050_180c93627d7b.slice/crio-b10446d8b4725941c12824c41ed5e992fcac26e0abaaa3e8ecc64f54391cb75c WatchSource:0}: Error finding container b10446d8b4725941c12824c41ed5e992fcac26e0abaaa3e8ecc64f54391cb75c: Status 404 returned error can't find the container with id b10446d8b4725941c12824c41ed5e992fcac26e0abaaa3e8ecc64f54391cb75c
Jul 16 22:22:51.186160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:51.186081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerStarted","Data":"1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898"}
Jul 16 22:22:51.186160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:51.186136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerStarted","Data":"b10446d8b4725941c12824c41ed5e992fcac26e0abaaa3e8ecc64f54391cb75c"}
Jul 16 22:22:51.233127 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:51.233076    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" podStartSLOduration=1.233061064 podStartE2EDuration="1.233061064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:51.231707564 +0000 UTC m=+2026.068761400" watchObservedRunningTime="2026-07-16 22:22:51.233061064 +0000 UTC m=+2026.070114863"
Jul 16 22:22:51.347864 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:51.347832    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e668468_6933_4cd2_9050_180c93627d7b.slice/crio-conmon-1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:22:52.192867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:52.192829    2555 generic.go:356] "Generic (PLEG): container finished" podID="2e668468-6933-4cd2-9050-180c93627d7b" containerID="1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898" exitCode=7
Jul 16 22:22:52.193046 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:52.192882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerDied","Data":"1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898"}
Jul 16 22:22:52.193151 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:52.193138    2555 scope.go:117] "RemoveContainer" containerID="1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898"
Jul 16 22:22:53.200593 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:53.200505    2555 generic.go:356] "Generic (PLEG): container finished" podID="2e668468-6933-4cd2-9050-180c93627d7b" containerID="5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8" exitCode=7
Jul 16 22:22:53.200992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:53.200595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerDied","Data":"5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8"}
Jul 16 22:22:53.200992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:53.200694    2555 scope.go:117] "RemoveContainer" containerID="1edf72c84e8b4cfc9837b73eb679bcb30d78357dff45b8195e9214e1d36f5898"
Jul 16 22:22:53.200992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:53.200877    2555 scope.go:117] "RemoveContainer" containerID="5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8"
Jul 16 22:22:53.201157 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:53.201078    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-mq6vh_odh-ai-gateway-infra(2e668468-6933-4cd2-9050-180c93627d7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" podUID="2e668468-6933-4cd2-9050-180c93627d7b"
Jul 16 22:22:54.208064 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:54.208036    2555 scope.go:117] "RemoveContainer" containerID="5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8"
Jul 16 22:22:54.208453 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:22:54.208244    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-mq6vh_odh-ai-gateway-infra(2e668468-6933-4cd2-9050-180c93627d7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" podUID="2e668468-6933-4cd2-9050-180c93627d7b"
Jul 16 22:22:59.706932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.706867    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:22:59.707378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.707289    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:59.707378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.707303    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:59.707378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.707368    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="93621d81-8c2f-4adc-96b0-8e468bd25a30" containerName="revoke-keys"
Jul 16 22:22:59.711636 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.711621    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:22:59.717555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.717526    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:22:59.848437 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.848396    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7\" (UID: \"dc47532d-80be-4a4b-a9f0-528fbff4d989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:22:59.948837 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.948805    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7\" (UID: \"dc47532d-80be-4a4b-a9f0-528fbff4d989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:22:59.949452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:22:59.949430    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7\" (UID: \"dc47532d-80be-4a4b-a9f0-528fbff4d989\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:23:00.023755 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:00.023665    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:23:00.166414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:00.166391    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:23:00.168244 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:23:00.168214    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc47532d_80be_4a4b_a9f0_528fbff4d989.slice/crio-f55d14671a002f0a917118cbff64d4e904dd74d310b9c4d33d522debad76ea3f WatchSource:0}: Error finding container f55d14671a002f0a917118cbff64d4e904dd74d310b9c4d33d522debad76ea3f: Status 404 returned error can't find the container with id f55d14671a002f0a917118cbff64d4e904dd74d310b9c4d33d522debad76ea3f
Jul 16 22:23:00.240682 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:00.240655    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" event={"ID":"dc47532d-80be-4a4b-a9f0-528fbff4d989","Type":"ContainerStarted","Data":"f55d14671a002f0a917118cbff64d4e904dd74d310b9c4d33d522debad76ea3f"}
Jul 16 22:23:01.247497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:01.247392    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerID="a5932e5bd3da5182c1de864409a4ce33047684d1dad97d0620cbf5c7de6ab298" exitCode=6
Jul 16 22:23:01.247913 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:01.247494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" event={"ID":"dc47532d-80be-4a4b-a9f0-528fbff4d989","Type":"ContainerDied","Data":"a5932e5bd3da5182c1de864409a4ce33047684d1dad97d0620cbf5c7de6ab298"}
Jul 16 22:23:01.247913 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:01.247672    2555 scope.go:117] "RemoveContainer" containerID="a5932e5bd3da5182c1de864409a4ce33047684d1dad97d0620cbf5c7de6ab298"
Jul 16 22:23:02.254252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:02.254165    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerID="9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31" exitCode=6
Jul 16 22:23:02.254687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:02.254253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" event={"ID":"dc47532d-80be-4a4b-a9f0-528fbff4d989","Type":"ContainerDied","Data":"9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31"}
Jul 16 22:23:02.254687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:02.254317    2555 scope.go:117] "RemoveContainer" containerID="a5932e5bd3da5182c1de864409a4ce33047684d1dad97d0620cbf5c7de6ab298"
Jul 16 22:23:02.254687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:02.254446    2555 scope.go:117] "RemoveContainer" containerID="9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31"
Jul 16 22:23:02.254813 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:02.254683    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7_odh-ai-gateway-infra(dc47532d-80be-4a4b-a9f0-528fbff4d989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989"
Jul 16 22:23:03.269278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:03.269252    2555 scope.go:117] "RemoveContainer" containerID="9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31"
Jul 16 22:23:03.269670 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:03.269428    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7_odh-ai-gateway-infra(dc47532d-80be-4a4b-a9f0-528fbff4d989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989"
Jul 16 22:23:05.673650 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:05.673622    2555 scope.go:117] "RemoveContainer" containerID="5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8"
Jul 16 22:23:06.286223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:06.286187    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerStarted","Data":"12278aa8d925c1e53c9a6896c16ab8a81dcc976a821aabec945c370440ca949a"}
Jul 16 22:23:07.293898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:07.293866    2555 generic.go:356] "Generic (PLEG): container finished" podID="2e668468-6933-4cd2-9050-180c93627d7b" containerID="12278aa8d925c1e53c9a6896c16ab8a81dcc976a821aabec945c370440ca949a" exitCode=7
Jul 16 22:23:07.294376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:07.293940    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerDied","Data":"12278aa8d925c1e53c9a6896c16ab8a81dcc976a821aabec945c370440ca949a"}
Jul 16 22:23:07.294376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:07.293996    2555 scope.go:117] "RemoveContainer" containerID="5425475be9b44458a50e19b87d230853f24bb3673d63cdfc94386e9893f8f1e8"
Jul 16 22:23:07.294376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:07.294273    2555 scope.go:117] "RemoveContainer" containerID="12278aa8d925c1e53c9a6896c16ab8a81dcc976a821aabec945c370440ca949a"
Jul 16 22:23:07.294570 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:07.294519    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-mq6vh_odh-ai-gateway-infra(2e668468-6933-4cd2-9050-180c93627d7b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" podUID="2e668468-6933-4cd2-9050-180c93627d7b"
Jul 16 22:23:07.321776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:07.321746    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:23:08.457886 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:08.457860    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:23:08.523414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:08.523381    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca\") pod \"2e668468-6933-4cd2-9050-180c93627d7b\" (UID: \"2e668468-6933-4cd2-9050-180c93627d7b\") "
Jul 16 22:23:08.523745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:08.523723    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2e668468-6933-4cd2-9050-180c93627d7b" (UID: "2e668468-6933-4cd2-9050-180c93627d7b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:08.624698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:08.624620    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2e668468-6933-4cd2-9050-180c93627d7b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:23:09.312877 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.312847    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"
Jul 16 22:23:09.312877 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.312863    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh" event={"ID":"2e668468-6933-4cd2-9050-180c93627d7b","Type":"ContainerDied","Data":"b10446d8b4725941c12824c41ed5e992fcac26e0abaaa3e8ecc64f54391cb75c"}
Jul 16 22:23:09.313083 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.312930    2555 scope.go:117] "RemoveContainer" containerID="12278aa8d925c1e53c9a6896c16ab8a81dcc976a821aabec945c370440ca949a"
Jul 16 22:23:09.338145 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.338115    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:23:09.344689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.344663    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-mq6vh"]
Jul 16 22:23:09.676485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:09.676397    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2e668468-6933-4cd2-9050-180c93627d7b" path="/var/lib/kubelet/pods/2e668468-6933-4cd2-9050-180c93627d7b/volumes"
Jul 16 22:23:14.670688 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:14.670655    2555 scope.go:117] "RemoveContainer" containerID="9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31"
Jul 16 22:23:15.345729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:15.345691    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerID="4b2bb56d093d1b34cf0269ce03c05ffdcd2d20281e9a139bf5f1cf1e0154656c" exitCode=6
Jul 16 22:23:15.345896 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:15.345767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" event={"ID":"dc47532d-80be-4a4b-a9f0-528fbff4d989","Type":"ContainerDied","Data":"4b2bb56d093d1b34cf0269ce03c05ffdcd2d20281e9a139bf5f1cf1e0154656c"}
Jul 16 22:23:15.345896 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:15.345864    2555 scope.go:117] "RemoveContainer" containerID="9a5e0201bc8332940a14f58f96b375d63bfd1f53bc9d0121d31bff8903c5de31"
Jul 16 22:23:15.346204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:15.346187    2555 scope.go:117] "RemoveContainer" containerID="4b2bb56d093d1b34cf0269ce03c05ffdcd2d20281e9a139bf5f1cf1e0154656c"
Jul 16 22:23:15.346525 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:15.346497    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7_odh-ai-gateway-infra(dc47532d-80be-4a4b-a9f0-528fbff4d989)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989"
Jul 16 22:23:16.372789 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:16.372754    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:23:16.506854 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:16.506833    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:23:16.599142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:16.599109    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca\") pod \"dc47532d-80be-4a4b-a9f0-528fbff4d989\" (UID: \"dc47532d-80be-4a4b-a9f0-528fbff4d989\") "
Jul 16 22:23:16.599532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:16.599508    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc47532d-80be-4a4b-a9f0-528fbff4d989" (UID: "dc47532d-80be-4a4b-a9f0-528fbff4d989"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:16.700779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:16.700687    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc47532d-80be-4a4b-a9f0-528fbff4d989-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:23:17.358155 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.358125    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"
Jul 16 22:23:17.358361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.358171    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7" event={"ID":"dc47532d-80be-4a4b-a9f0-528fbff4d989","Type":"ContainerDied","Data":"f55d14671a002f0a917118cbff64d4e904dd74d310b9c4d33d522debad76ea3f"}
Jul 16 22:23:17.358361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.358228    2555 scope.go:117] "RemoveContainer" containerID="4b2bb56d093d1b34cf0269ce03c05ffdcd2d20281e9a139bf5f1cf1e0154656c"
Jul 16 22:23:17.380519 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.380487    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:23:17.387173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.387150    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-lkwj7"]
Jul 16 22:23:17.675860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:17.675779    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" path="/var/lib/kubelet/pods/dc47532d-80be-4a4b-a9f0-528fbff4d989/volumes"
Jul 16 22:23:18.517116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517083    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517419    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517430    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517437    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517442    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517450    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517456    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517487    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517492    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517501    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517506    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517512    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517518    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517571    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517579    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517586    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:18.517894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517592    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.517894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.517598    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc47532d-80be-4a4b-a9f0-528fbff4d989" containerName="revoke-keys"
Jul 16 22:23:18.522037 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.522019    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:18.524539 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.524519    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:23:18.524638 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.524551    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:23:18.529686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.529666    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:18.616992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.616948    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz\" (UID: \"c03592e8-e948-4544-8988-b8bd23c64958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:18.717810 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.717780    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz\" (UID: \"c03592e8-e948-4544-8988-b8bd23c64958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:18.718412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.718393    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz\" (UID: \"c03592e8-e948-4544-8988-b8bd23c64958\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:18.834256 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.834224    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:18.969202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:18.969180    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:18.971241 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:23:18.971214    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc03592e8_e948_4544_8988_b8bd23c64958.slice/crio-450b614b1d7dd8ff58ff8f6be6fd7300b2aea7c75f05e79728151990069c953d WatchSource:0}: Error finding container 450b614b1d7dd8ff58ff8f6be6fd7300b2aea7c75f05e79728151990069c953d: Status 404 returned error can't find the container with id 450b614b1d7dd8ff58ff8f6be6fd7300b2aea7c75f05e79728151990069c953d
Jul 16 22:23:19.373034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:19.372246    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerStarted","Data":"fc5f8fd87c740b8be73ca7fec4b5eef78fcc9b336b4dc4be2bc1cd2f2032cba1"}
Jul 16 22:23:19.373034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:19.372315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerStarted","Data":"450b614b1d7dd8ff58ff8f6be6fd7300b2aea7c75f05e79728151990069c953d"}
Jul 16 22:23:19.396509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:19.396431    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" podStartSLOduration=1.396415404 podStartE2EDuration="1.396415404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:19.387523978 +0000 UTC m=+2054.224577781" watchObservedRunningTime="2026-07-16 22:23:19.396415404 +0000 UTC m=+2054.233469204"
Jul 16 22:23:20.378723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:20.378684    2555 generic.go:356] "Generic (PLEG): container finished" podID="c03592e8-e948-4544-8988-b8bd23c64958" containerID="fc5f8fd87c740b8be73ca7fec4b5eef78fcc9b336b4dc4be2bc1cd2f2032cba1" exitCode=6
Jul 16 22:23:20.379117 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:20.378769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerDied","Data":"fc5f8fd87c740b8be73ca7fec4b5eef78fcc9b336b4dc4be2bc1cd2f2032cba1"}
Jul 16 22:23:20.379117 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:20.378915    2555 scope.go:117] "RemoveContainer" containerID="fc5f8fd87c740b8be73ca7fec4b5eef78fcc9b336b4dc4be2bc1cd2f2032cba1"
Jul 16 22:23:21.385797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:21.385764    2555 generic.go:356] "Generic (PLEG): container finished" podID="c03592e8-e948-4544-8988-b8bd23c64958" containerID="17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a" exitCode=6
Jul 16 22:23:21.386240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:21.385862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerDied","Data":"17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a"}
Jul 16 22:23:21.386240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:21.385930    2555 scope.go:117] "RemoveContainer" containerID="fc5f8fd87c740b8be73ca7fec4b5eef78fcc9b336b4dc4be2bc1cd2f2032cba1"
Jul 16 22:23:21.386240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:21.386059    2555 scope.go:117] "RemoveContainer" containerID="17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a"
Jul 16 22:23:21.386440 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:21.386286    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz_odh-ai-gateway-infra(c03592e8-e948-4544-8988-b8bd23c64958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" podUID="c03592e8-e948-4544-8988-b8bd23c64958"
Jul 16 22:23:22.392635 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:22.392606    2555 scope.go:117] "RemoveContainer" containerID="17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a"
Jul 16 22:23:22.393039 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:22.392775    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz_odh-ai-gateway-infra(c03592e8-e948-4544-8988-b8bd23c64958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" podUID="c03592e8-e948-4544-8988-b8bd23c64958"
Jul 16 22:23:35.673568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:35.673488    2555 scope.go:117] "RemoveContainer" containerID="17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a"
Jul 16 22:23:36.464553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:36.464526    2555 generic.go:356] "Generic (PLEG): container finished" podID="c03592e8-e948-4544-8988-b8bd23c64958" containerID="8e94843fb76101a6961ef64ff54e8895527a1f5d5f5b30ec80d5deade5af7db0" exitCode=6
Jul 16 22:23:36.464684 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:36.464611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerDied","Data":"8e94843fb76101a6961ef64ff54e8895527a1f5d5f5b30ec80d5deade5af7db0"}
Jul 16 22:23:36.464684 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:36.464677    2555 scope.go:117] "RemoveContainer" containerID="17a054ab08cdc0364f47e16d11f9910563b2c440f3b14ce44abaaa321715e99a"
Jul 16 22:23:36.464943 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:36.464927    2555 scope.go:117] "RemoveContainer" containerID="8e94843fb76101a6961ef64ff54e8895527a1f5d5f5b30ec80d5deade5af7db0"
Jul 16 22:23:36.465211 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:36.465190    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz_odh-ai-gateway-infra(c03592e8-e948-4544-8988-b8bd23c64958)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" podUID="c03592e8-e948-4544-8988-b8bd23c64958"
Jul 16 22:23:37.493516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:37.493484    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:37.626755 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:37.626729    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:37.666643 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:37.666615    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca\") pod \"c03592e8-e948-4544-8988-b8bd23c64958\" (UID: \"c03592e8-e948-4544-8988-b8bd23c64958\") "
Jul 16 22:23:37.666941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:37.666918    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c03592e8-e948-4544-8988-b8bd23c64958" (UID: "c03592e8-e948-4544-8988-b8bd23c64958"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:37.767761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:37.767673    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c03592e8-e948-4544-8988-b8bd23c64958-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:23:38.476856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:38.476820    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz" event={"ID":"c03592e8-e948-4544-8988-b8bd23c64958","Type":"ContainerDied","Data":"450b614b1d7dd8ff58ff8f6be6fd7300b2aea7c75f05e79728151990069c953d"}
Jul 16 22:23:38.477045 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:38.476886    2555 scope.go:117] "RemoveContainer" containerID="8e94843fb76101a6961ef64ff54e8895527a1f5d5f5b30ec80d5deade5af7db0"
Jul 16 22:23:38.477045 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:38.476827    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"
Jul 16 22:23:38.493508 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:38.493481    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:38.498047 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:38.498026    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-4r9rz"]
Jul 16 22:23:39.676853 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:39.676819    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c03592e8-e948-4544-8988-b8bd23c64958" path="/var/lib/kubelet/pods/c03592e8-e948-4544-8988-b8bd23c64958/volumes"
Jul 16 22:23:40.273644 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.273613    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:40.274055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274038    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274059    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274070    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274078    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274110    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274160 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274118    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274211    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274223    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:40.274398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.274238    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2e668468-6933-4cd2-9050-180c93627d7b" containerName="revoke-keys"
Jul 16 22:23:40.278647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.278628    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:40.281175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.281145    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:23:40.281276 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.281244    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:23:40.285109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.285091    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:40.287723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.287695    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-grbnc\" (UID: \"ac9a8454-3daa-4434-bafe-89a31bab77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:40.389018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.388985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-grbnc\" (UID: \"ac9a8454-3daa-4434-bafe-89a31bab77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:40.389583 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.389565    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-grbnc\" (UID: \"ac9a8454-3daa-4434-bafe-89a31bab77b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:40.590578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.590533    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:40.735673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:40.735650    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:40.739952 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:23:40.739922    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac9a8454_3daa_4434_bafe_89a31bab77b8.slice/crio-45b052a653724fdb6e660f6f631e4da4cbf15cc7d0ffb45b1772bfb2ee8a4438 WatchSource:0}: Error finding container 45b052a653724fdb6e660f6f631e4da4cbf15cc7d0ffb45b1772bfb2ee8a4438: Status 404 returned error can't find the container with id 45b052a653724fdb6e660f6f631e4da4cbf15cc7d0ffb45b1772bfb2ee8a4438
Jul 16 22:23:41.495623 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:41.495591    2555 generic.go:356] "Generic (PLEG): container finished" podID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerID="422f313b8d7a0526ea3abb0669e5c0859b1d2743c268511357e527a94caa3869" exitCode=7
Jul 16 22:23:41.495791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:41.495675    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" event={"ID":"ac9a8454-3daa-4434-bafe-89a31bab77b8","Type":"ContainerDied","Data":"422f313b8d7a0526ea3abb0669e5c0859b1d2743c268511357e527a94caa3869"}
Jul 16 22:23:41.495791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:41.495729    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" event={"ID":"ac9a8454-3daa-4434-bafe-89a31bab77b8","Type":"ContainerStarted","Data":"45b052a653724fdb6e660f6f631e4da4cbf15cc7d0ffb45b1772bfb2ee8a4438"}
Jul 16 22:23:41.495872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:41.495802    2555 scope.go:117] "RemoveContainer" containerID="422f313b8d7a0526ea3abb0669e5c0859b1d2743c268511357e527a94caa3869"
Jul 16 22:23:42.503894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:42.503800    2555 generic.go:356] "Generic (PLEG): container finished" podID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerID="5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0" exitCode=7
Jul 16 22:23:42.504332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:42.503886    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" event={"ID":"ac9a8454-3daa-4434-bafe-89a31bab77b8","Type":"ContainerDied","Data":"5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0"}
Jul 16 22:23:42.504332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:42.503959    2555 scope.go:117] "RemoveContainer" containerID="422f313b8d7a0526ea3abb0669e5c0859b1d2743c268511357e527a94caa3869"
Jul 16 22:23:42.504332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:42.504162    2555 scope.go:117] "RemoveContainer" containerID="5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0"
Jul 16 22:23:42.504488 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:42.504385    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-grbnc_odh-ai-gateway-infra(ac9a8454-3daa-4434-bafe-89a31bab77b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8"
Jul 16 22:23:43.510963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:43.510936    2555 scope.go:117] "RemoveContainer" containerID="5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0"
Jul 16 22:23:43.511354 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:43.511119    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-grbnc_odh-ai-gateway-infra(ac9a8454-3daa-4434-bafe-89a31bab77b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8"
Jul 16 22:23:49.749856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.749820    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:23:49.750450 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.750428    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c03592e8-e948-4544-8988-b8bd23c64958" containerName="revoke-keys"
Jul 16 22:23:49.754970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.754946    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:23:49.760990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.760944    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:23:49.876032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.875997    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b\" (UID: \"a02aa05d-b600-4ba4-8476-2ee93d074005\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:23:49.977179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.977145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b\" (UID: \"a02aa05d-b600-4ba4-8476-2ee93d074005\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:23:49.977859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:49.977837    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b\" (UID: \"a02aa05d-b600-4ba4-8476-2ee93d074005\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:23:50.068963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:50.068934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:23:50.207176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:50.207150    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:23:50.209239 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:23:50.209213    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda02aa05d_b600_4ba4_8476_2ee93d074005.slice/crio-571cf7e5e4bc641452a70756a06c1716787b4e29cfeccc44abb44d83df620310 WatchSource:0}: Error finding container 571cf7e5e4bc641452a70756a06c1716787b4e29cfeccc44abb44d83df620310: Status 404 returned error can't find the container with id 571cf7e5e4bc641452a70756a06c1716787b4e29cfeccc44abb44d83df620310
Jul 16 22:23:50.550742 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:50.550705    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerStarted","Data":"f33b08930cdf3805e93c919f2523dcaf2351662ed8985fc34f95a9a90081dc15"}
Jul 16 22:23:50.550902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:50.550771    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerStarted","Data":"571cf7e5e4bc641452a70756a06c1716787b4e29cfeccc44abb44d83df620310"}
Jul 16 22:23:50.568793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:50.568747    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" podStartSLOduration=1.568732656 podStartE2EDuration="1.568732656s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:50.567831354 +0000 UTC m=+2085.404885153" watchObservedRunningTime="2026-07-16 22:23:50.568732656 +0000 UTC m=+2085.405786455"
Jul 16 22:23:51.557745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:51.557711    2555 generic.go:356] "Generic (PLEG): container finished" podID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerID="f33b08930cdf3805e93c919f2523dcaf2351662ed8985fc34f95a9a90081dc15" exitCode=6
Jul 16 22:23:51.558141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:51.557784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerDied","Data":"f33b08930cdf3805e93c919f2523dcaf2351662ed8985fc34f95a9a90081dc15"}
Jul 16 22:23:51.558141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:51.558005    2555 scope.go:117] "RemoveContainer" containerID="f33b08930cdf3805e93c919f2523dcaf2351662ed8985fc34f95a9a90081dc15"
Jul 16 22:23:52.565345 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:52.565309    2555 generic.go:356] "Generic (PLEG): container finished" podID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerID="6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8" exitCode=6
Jul 16 22:23:52.565822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:52.565359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerDied","Data":"6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8"}
Jul 16 22:23:52.565822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:52.565410    2555 scope.go:117] "RemoveContainer" containerID="f33b08930cdf3805e93c919f2523dcaf2351662ed8985fc34f95a9a90081dc15"
Jul 16 22:23:52.565822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:52.565654    2555 scope.go:117] "RemoveContainer" containerID="6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8"
Jul 16 22:23:52.565992 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:52.565924    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b_odh-ai-gateway-infra(a02aa05d-b600-4ba4-8476-2ee93d074005)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005"
Jul 16 22:23:53.579077 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:53.579040    2555 scope.go:117] "RemoveContainer" containerID="6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8"
Jul 16 22:23:53.579596 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:53.579355    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b_odh-ai-gateway-infra(a02aa05d-b600-4ba4-8476-2ee93d074005)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005"
Jul 16 22:23:54.671134 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:54.671105    2555 scope.go:117] "RemoveContainer" containerID="5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0"
Jul 16 22:23:55.590116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:55.590084    2555 generic.go:356] "Generic (PLEG): container finished" podID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerID="e3d170c5aebc7333efe0137f98bb1e94ad929991b99bb990daaf99c3df12194f" exitCode=7
Jul 16 22:23:55.590313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:55.590165    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" event={"ID":"ac9a8454-3daa-4434-bafe-89a31bab77b8","Type":"ContainerDied","Data":"e3d170c5aebc7333efe0137f98bb1e94ad929991b99bb990daaf99c3df12194f"}
Jul 16 22:23:55.590313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:55.590248    2555 scope.go:117] "RemoveContainer" containerID="5d923b5b14389789ef090ce0b97b74c7d172863a826eaffb0764ab1edc9b78a0"
Jul 16 22:23:55.590564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:55.590543    2555 scope.go:117] "RemoveContainer" containerID="e3d170c5aebc7333efe0137f98bb1e94ad929991b99bb990daaf99c3df12194f"
Jul 16 22:23:55.590773 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:23:55.590751    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-grbnc_odh-ai-gateway-infra(ac9a8454-3daa-4434-bafe-89a31bab77b8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8"
Jul 16 22:23:56.623228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:56.623196    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:56.758207 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:56.758184    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:56.838295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:56.838269    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca\") pod \"ac9a8454-3daa-4434-bafe-89a31bab77b8\" (UID: \"ac9a8454-3daa-4434-bafe-89a31bab77b8\") "
Jul 16 22:23:56.838622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:56.838603    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ac9a8454-3daa-4434-bafe-89a31bab77b8" (UID: "ac9a8454-3daa-4434-bafe-89a31bab77b8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:56.939520 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:56.939405    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac9a8454-3daa-4434-bafe-89a31bab77b8-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:23:57.604819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.604783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc" event={"ID":"ac9a8454-3daa-4434-bafe-89a31bab77b8","Type":"ContainerDied","Data":"45b052a653724fdb6e660f6f631e4da4cbf15cc7d0ffb45b1772bfb2ee8a4438"}
Jul 16 22:23:57.604819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.604806    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"
Jul 16 22:23:57.605061 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.604861    2555 scope.go:117] "RemoveContainer" containerID="e3d170c5aebc7333efe0137f98bb1e94ad929991b99bb990daaf99c3df12194f"
Jul 16 22:23:57.625828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.625804    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:57.629813 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.629789    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-grbnc"]
Jul 16 22:23:57.675807 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:23:57.675764    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" path="/var/lib/kubelet/pods/ac9a8454-3daa-4434-bafe-89a31bab77b8/volumes"
Jul 16 22:24:06.670411 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:06.670383    2555 scope.go:117] "RemoveContainer" containerID="6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8"
Jul 16 22:24:07.659728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:07.659643    2555 generic.go:356] "Generic (PLEG): container finished" podID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerID="0fa8c9bc23b17c6587b7f1e7b6b4d625a630a7dee36cc00ae84075b088a1d907" exitCode=6
Jul 16 22:24:07.659728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:07.659714    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerDied","Data":"0fa8c9bc23b17c6587b7f1e7b6b4d625a630a7dee36cc00ae84075b088a1d907"}
Jul 16 22:24:07.659915 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:07.659775    2555 scope.go:117] "RemoveContainer" containerID="6bc961f01f02ae3fe491ee0da1991af371af491b527cd74b53964136f7df9ba8"
Jul 16 22:24:07.660042 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:07.660027    2555 scope.go:117] "RemoveContainer" containerID="0fa8c9bc23b17c6587b7f1e7b6b4d625a630a7dee36cc00ae84075b088a1d907"
Jul 16 22:24:07.660277 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:07.660256    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b_odh-ai-gateway-infra(a02aa05d-b600-4ba4-8476-2ee93d074005)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005"
Jul 16 22:24:08.689244 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:08.689205    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:24:08.821607 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:08.821583    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:24:08.946778 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:08.946685    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca\") pod \"a02aa05d-b600-4ba4-8476-2ee93d074005\" (UID: \"a02aa05d-b600-4ba4-8476-2ee93d074005\") "
Jul 16 22:24:08.947056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:08.947031    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a02aa05d-b600-4ba4-8476-2ee93d074005" (UID: "a02aa05d-b600-4ba4-8476-2ee93d074005"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:09.047282 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.047234    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a02aa05d-b600-4ba4-8476-2ee93d074005-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:24:09.673158 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.673131    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"
Jul 16 22:24:09.676020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.675979    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b" event={"ID":"a02aa05d-b600-4ba4-8476-2ee93d074005","Type":"ContainerDied","Data":"571cf7e5e4bc641452a70756a06c1716787b4e29cfeccc44abb44d83df620310"}
Jul 16 22:24:09.676157 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.676031    2555 scope.go:117] "RemoveContainer" containerID="0fa8c9bc23b17c6587b7f1e7b6b4d625a630a7dee36cc00ae84075b088a1d907"
Jul 16 22:24:09.705943 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.705904    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:24:09.712119 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:09.712097    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-22s6b"]
Jul 16 22:24:11.675700 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:11.675667    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" path="/var/lib/kubelet/pods/a02aa05d-b600-4ba4-8476-2ee93d074005/volumes"
Jul 16 22:24:18.591591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591559    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591914    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591926    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591944    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591954    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591965    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591970    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591981    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.591986    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592000    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592006    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592071    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592079    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592087    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592094    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.592217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.592099    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a02aa05d-b600-4ba4-8476-2ee93d074005" containerName="revoke-keys"
Jul 16 22:24:18.596306 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.596287    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:18.599812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.599792    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:24:18.600369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.600352    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:24:18.613060 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.613036    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:18.733038 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.733007    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj\" (UID: \"a75574b7-38b5-4d81-a116-8a3331ee09ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:18.834049 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.834010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj\" (UID: \"a75574b7-38b5-4d81-a116-8a3331ee09ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:18.834749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.834724    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj\" (UID: \"a75574b7-38b5-4d81-a116-8a3331ee09ed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:18.906305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:18.906214    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:19.053996 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:19.053970    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:19.056184 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:24:19.056155    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75574b7_38b5_4d81_a116_8a3331ee09ed.slice/crio-95cd6c1815c46e53a861e344fb845b4945a9acfffabd5afe9a60981358618165 WatchSource:0}: Error finding container 95cd6c1815c46e53a861e344fb845b4945a9acfffabd5afe9a60981358618165: Status 404 returned error can't find the container with id 95cd6c1815c46e53a861e344fb845b4945a9acfffabd5afe9a60981358618165
Jul 16 22:24:19.725278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:19.725245    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerID="db85c257258e47ccb2facfc7e0eec5e7e15322b32dd3775b14b8d056af826aaa" exitCode=6
Jul 16 22:24:19.725699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:19.725309    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" event={"ID":"a75574b7-38b5-4d81-a116-8a3331ee09ed","Type":"ContainerDied","Data":"db85c257258e47ccb2facfc7e0eec5e7e15322b32dd3775b14b8d056af826aaa"}
Jul 16 22:24:19.725699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:19.725347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" event={"ID":"a75574b7-38b5-4d81-a116-8a3331ee09ed","Type":"ContainerStarted","Data":"95cd6c1815c46e53a861e344fb845b4945a9acfffabd5afe9a60981358618165"}
Jul 16 22:24:19.725699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:19.725556    2555 scope.go:117] "RemoveContainer" containerID="db85c257258e47ccb2facfc7e0eec5e7e15322b32dd3775b14b8d056af826aaa"
Jul 16 22:24:20.732740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:20.732660    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerID="c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01" exitCode=6
Jul 16 22:24:20.732740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:20.732710    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" event={"ID":"a75574b7-38b5-4d81-a116-8a3331ee09ed","Type":"ContainerDied","Data":"c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01"}
Jul 16 22:24:20.733284 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:20.732757    2555 scope.go:117] "RemoveContainer" containerID="db85c257258e47ccb2facfc7e0eec5e7e15322b32dd3775b14b8d056af826aaa"
Jul 16 22:24:20.733284 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:20.732935    2555 scope.go:117] "RemoveContainer" containerID="c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01"
Jul 16 22:24:20.733284 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:20.733140    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj_odh-ai-gateway-infra(a75574b7-38b5-4d81-a116-8a3331ee09ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed"
Jul 16 22:24:21.740046 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:21.740015    2555 scope.go:117] "RemoveContainer" containerID="c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01"
Jul 16 22:24:21.740562 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:21.740298    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj_odh-ai-gateway-infra(a75574b7-38b5-4d81-a116-8a3331ee09ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed"
Jul 16 22:24:30.341629 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.341587    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:30.344092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.342064    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:30.344092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.342079    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:30.344092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.342165    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac9a8454-3daa-4434-bafe-89a31bab77b8" containerName="revoke-keys"
Jul 16 22:24:30.345187 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.345168    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:30.354289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.354265    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:30.435043 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.435004    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-pvfph\" (UID: \"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:30.536380 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.536340    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-pvfph\" (UID: \"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:30.536947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.536928    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-pvfph\" (UID: \"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:30.656501 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.656389    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:30.793082 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:30.793057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:31.791546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:31.791518    2555 generic.go:356] "Generic (PLEG): container finished" podID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerID="f4df0d1efd74b49f252ae90de86d0ed3e1e17920693e3d345ce385c9fc88d273" exitCode=7
Jul 16 22:24:31.791982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:31.791611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" event={"ID":"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2","Type":"ContainerDied","Data":"f4df0d1efd74b49f252ae90de86d0ed3e1e17920693e3d345ce385c9fc88d273"}
Jul 16 22:24:31.791982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:31.791658    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" event={"ID":"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2","Type":"ContainerStarted","Data":"9eb0d7ba9ec99df08d216bafaa0c2997fa6dd755a6ae77e746dfae042578c109"}
Jul 16 22:24:31.791982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:31.791790    2555 scope.go:117] "RemoveContainer" containerID="f4df0d1efd74b49f252ae90de86d0ed3e1e17920693e3d345ce385c9fc88d273"
Jul 16 22:24:32.804310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:32.804277    2555 generic.go:356] "Generic (PLEG): container finished" podID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerID="c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5" exitCode=7
Jul 16 22:24:32.804726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:32.804361    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" event={"ID":"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2","Type":"ContainerDied","Data":"c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5"}
Jul 16 22:24:32.804726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:32.804449    2555 scope.go:117] "RemoveContainer" containerID="f4df0d1efd74b49f252ae90de86d0ed3e1e17920693e3d345ce385c9fc88d273"
Jul 16 22:24:32.804726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:32.804688    2555 scope.go:117] "RemoveContainer" containerID="c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5"
Jul 16 22:24:32.804914 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:32.804895    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-pvfph_odh-ai-gateway-infra(7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2"
Jul 16 22:24:33.810610 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:33.810584    2555 scope.go:117] "RemoveContainer" containerID="c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5"
Jul 16 22:24:33.810989 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:33.810777    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-pvfph_odh-ai-gateway-infra(7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2"
Jul 16 22:24:35.674070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:35.674041    2555 scope.go:117] "RemoveContainer" containerID="c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01"
Jul 16 22:24:36.827853 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:36.827820    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerID="413329cb22f4b2b86e2fae5952f65ecb35434b0d4c448341cb7193178ed18bd4" exitCode=6
Jul 16 22:24:36.828243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:36.827876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" event={"ID":"a75574b7-38b5-4d81-a116-8a3331ee09ed","Type":"ContainerDied","Data":"413329cb22f4b2b86e2fae5952f65ecb35434b0d4c448341cb7193178ed18bd4"}
Jul 16 22:24:36.828243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:36.827926    2555 scope.go:117] "RemoveContainer" containerID="c7f92408c3870ec841a2977102587462475ab893122aa709fb1a41a34ca66a01"
Jul 16 22:24:36.828243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:36.828231    2555 scope.go:117] "RemoveContainer" containerID="413329cb22f4b2b86e2fae5952f65ecb35434b0d4c448341cb7193178ed18bd4"
Jul 16 22:24:36.828477 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:36.828447    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj_odh-ai-gateway-infra(a75574b7-38b5-4d81-a116-8a3331ee09ed)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed"
Jul 16 22:24:37.863472 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:37.863439    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:37.986643 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:37.986616    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:38.103547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.103515    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca\") pod \"a75574b7-38b5-4d81-a116-8a3331ee09ed\" (UID: \"a75574b7-38b5-4d81-a116-8a3331ee09ed\") "
Jul 16 22:24:38.103883 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.103858    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a75574b7-38b5-4d81-a116-8a3331ee09ed" (UID: "a75574b7-38b5-4d81-a116-8a3331ee09ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:38.205147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.205053    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75574b7-38b5-4d81-a116-8a3331ee09ed-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:24:38.841478 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.841434    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"
Jul 16 22:24:38.841659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.841503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj" event={"ID":"a75574b7-38b5-4d81-a116-8a3331ee09ed","Type":"ContainerDied","Data":"95cd6c1815c46e53a861e344fb845b4945a9acfffabd5afe9a60981358618165"}
Jul 16 22:24:38.841659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.841566    2555 scope.go:117] "RemoveContainer" containerID="413329cb22f4b2b86e2fae5952f65ecb35434b0d4c448341cb7193178ed18bd4"
Jul 16 22:24:38.864026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.864000    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:38.868093 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:38.868070    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7dzfj"]
Jul 16 22:24:39.676821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.676786    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" path="/var/lib/kubelet/pods/a75574b7-38b5-4d81-a116-8a3331ee09ed/volumes"
Jul 16 22:24:39.804559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804529    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:39.804882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804870    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.804933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804884    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.804933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804893    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.804933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804898    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.804933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804918    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.804933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804924    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.805106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804981    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.805106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.804989    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:24:39.809319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.809302    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:39.815181 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.815156    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:39.919273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:39.919242    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf\" (UID: \"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:40.020759 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.020669    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf\" (UID: \"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:40.021358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.021340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf\" (UID: \"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:40.121560 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.121527    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:40.257280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.257240    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:40.260547 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:24:40.260510    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c8c030_8aee_4eb3_8bf7_fc8acad293ef.slice/crio-729f6986d3cf4b90be6361f27332fc5ff07fa102a7541ccafd363579b36e08ae WatchSource:0}: Error finding container 729f6986d3cf4b90be6361f27332fc5ff07fa102a7541ccafd363579b36e08ae: Status 404 returned error can't find the container with id 729f6986d3cf4b90be6361f27332fc5ff07fa102a7541ccafd363579b36e08ae
Jul 16 22:24:40.854928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.854839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerStarted","Data":"3de5507651e5f7538da749cdfcefa24075eecc43a0139f1d72597937a4a43c25"}
Jul 16 22:24:40.854928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.854891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerStarted","Data":"729f6986d3cf4b90be6361f27332fc5ff07fa102a7541ccafd363579b36e08ae"}
Jul 16 22:24:40.873165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:40.873115    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" podStartSLOduration=1.873101516 podStartE2EDuration="1.873101516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:24:40.871612138 +0000 UTC m=+2135.708665952" watchObservedRunningTime="2026-07-16 22:24:40.873101516 +0000 UTC m=+2135.710155315"
Jul 16 22:24:41.862037 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:41.862005    2555 generic.go:356] "Generic (PLEG): container finished" podID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerID="3de5507651e5f7538da749cdfcefa24075eecc43a0139f1d72597937a4a43c25" exitCode=6
Jul 16 22:24:41.862427 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:41.862136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerDied","Data":"3de5507651e5f7538da749cdfcefa24075eecc43a0139f1d72597937a4a43c25"}
Jul 16 22:24:41.862427 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:41.862246    2555 scope.go:117] "RemoveContainer" containerID="3de5507651e5f7538da749cdfcefa24075eecc43a0139f1d72597937a4a43c25"
Jul 16 22:24:42.870195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:42.870160    2555 generic.go:356] "Generic (PLEG): container finished" podID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerID="28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7" exitCode=6
Jul 16 22:24:42.870695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:42.870204    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerDied","Data":"28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7"}
Jul 16 22:24:42.870695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:42.870269    2555 scope.go:117] "RemoveContainer" containerID="3de5507651e5f7538da749cdfcefa24075eecc43a0139f1d72597937a4a43c25"
Jul 16 22:24:42.870695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:42.870420    2555 scope.go:117] "RemoveContainer" containerID="28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7"
Jul 16 22:24:42.870695 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:42.870672    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf_odh-ai-gateway-infra(d9c8c030-8aee-4eb3-8bf7-fc8acad293ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef"
Jul 16 22:24:43.877069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:43.877034    2555 scope.go:117] "RemoveContainer" containerID="28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7"
Jul 16 22:24:43.877568 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:43.877284    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf_odh-ai-gateway-infra(d9c8c030-8aee-4eb3-8bf7-fc8acad293ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef"
Jul 16 22:24:49.671190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:49.671152    2555 scope.go:117] "RemoveContainer" containerID="c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5"
Jul 16 22:24:50.912736 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:50.912703    2555 generic.go:356] "Generic (PLEG): container finished" podID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerID="1fd8fbcce356d7ecde9ef8f405ebc0805d36cb1c906aa045b6fdb366a0783564" exitCode=7
Jul 16 22:24:50.913154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:50.912775    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" event={"ID":"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2","Type":"ContainerDied","Data":"1fd8fbcce356d7ecde9ef8f405ebc0805d36cb1c906aa045b6fdb366a0783564"}
Jul 16 22:24:50.913154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:50.912862    2555 scope.go:117] "RemoveContainer" containerID="c3b32b7b83ae068b04ea1d9c5c6067988c8bb96835ba6715f2036c03d63644b5"
Jul 16 22:24:50.913154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:50.913140    2555 scope.go:117] "RemoveContainer" containerID="1fd8fbcce356d7ecde9ef8f405ebc0805d36cb1c906aa045b6fdb366a0783564"
Jul 16 22:24:50.913380 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:50.913358    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-pvfph_odh-ai-gateway-infra(7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2"
Jul 16 22:24:51.947373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:51.947341    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:52.085547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.085518    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:52.231867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.231772    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca\") pod \"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2\" (UID: \"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2\") "
Jul 16 22:24:52.232163 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.232138    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" (UID: "7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:52.332798 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.332739    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:24:52.925327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.925300    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"
Jul 16 22:24:52.925327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.925321    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph" event={"ID":"7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2","Type":"ContainerDied","Data":"9eb0d7ba9ec99df08d216bafaa0c2997fa6dd755a6ae77e746dfae042578c109"}
Jul 16 22:24:52.925558 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.925383    2555 scope.go:117] "RemoveContainer" containerID="1fd8fbcce356d7ecde9ef8f405ebc0805d36cb1c906aa045b6fdb366a0783564"
Jul 16 22:24:52.953677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.953651    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:52.958034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:52.958013    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-pvfph"]
Jul 16 22:24:53.677280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:53.677245    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" path="/var/lib/kubelet/pods/7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2/volumes"
Jul 16 22:24:55.677497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:55.677444    2555 scope.go:117] "RemoveContainer" containerID="28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7"
Jul 16 22:24:56.949141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:56.949107    2555 generic.go:356] "Generic (PLEG): container finished" podID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerID="54974471c711cfa710e1cc8260baabe44ed5945f4549836fac846302652d0e86" exitCode=6
Jul 16 22:24:56.949603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:56.949184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerDied","Data":"54974471c711cfa710e1cc8260baabe44ed5945f4549836fac846302652d0e86"}
Jul 16 22:24:56.949603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:56.949247    2555 scope.go:117] "RemoveContainer" containerID="28e8ab79f5a1408d15635a8fef16ee105d7e20b95109ee6322ba698e809d45f7"
Jul 16 22:24:56.949603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:56.949567    2555 scope.go:117] "RemoveContainer" containerID="54974471c711cfa710e1cc8260baabe44ed5945f4549836fac846302652d0e86"
Jul 16 22:24:56.949811 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:24:56.949791    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf_odh-ai-gateway-infra(d9c8c030-8aee-4eb3-8bf7-fc8acad293ef)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef"
Jul 16 22:24:57.977772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:57.977739    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:58.110035 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.110008    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:58.184996 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.184970    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca\") pod \"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef\" (UID: \"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef\") "
Jul 16 22:24:58.185358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.185336    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" (UID: "d9c8c030-8aee-4eb3-8bf7-fc8acad293ef"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:24:58.286320 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.286287    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:24:58.962631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.962602    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"
Jul 16 22:24:58.962793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.962649    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf" event={"ID":"d9c8c030-8aee-4eb3-8bf7-fc8acad293ef","Type":"ContainerDied","Data":"729f6986d3cf4b90be6361f27332fc5ff07fa102a7541ccafd363579b36e08ae"}
Jul 16 22:24:58.962793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.962706    2555 scope.go:117] "RemoveContainer" containerID="54974471c711cfa710e1cc8260baabe44ed5945f4549836fac846302652d0e86"
Jul 16 22:24:58.984398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.984368    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:58.990408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:58.990387    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8l5mf"]
Jul 16 22:24:59.676326 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:24:59.676294    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" path="/var/lib/kubelet/pods/d9c8c030-8aee-4eb3-8bf7-fc8acad293ef/volumes"
Jul 16 22:25:18.647526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.647483    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:18.648011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.647937    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.647957    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.647986    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.647995    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648015    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648049    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648065    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648075    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648152    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648165    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648175    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:18.648379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648191    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648199    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75574b7-38b5-4d81-a116-8a3331ee09ed" containerName="revoke-keys"
Jul 16 22:25:18.648379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648207    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.648379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.648216    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:18.651429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.651411    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:18.664616 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.664594    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:25:18.664887 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.664873    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:25:18.672033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.672006    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:18.751154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.751116    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc\" (UID: \"f87faf72-693b-4d4d-b605-e93309fb444b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:18.851601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.851563    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc\" (UID: \"f87faf72-693b-4d4d-b605-e93309fb444b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:18.852199 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.852171    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc\" (UID: \"f87faf72-693b-4d4d-b605-e93309fb444b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:18.961628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:18.961546    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:19.098514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:19.098455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:19.103680 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:25:19.103651    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf87faf72_693b_4d4d_b605_e93309fb444b.slice/crio-b5dc33e816ce406d1da4acd589b0fdb8219f00433649dc2c486ca34d5dc8a0f7 WatchSource:0}: Error finding container b5dc33e816ce406d1da4acd589b0fdb8219f00433649dc2c486ca34d5dc8a0f7: Status 404 returned error can't find the container with id b5dc33e816ce406d1da4acd589b0fdb8219f00433649dc2c486ca34d5dc8a0f7
Jul 16 22:25:20.074228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:20.074190    2555 generic.go:356] "Generic (PLEG): container finished" podID="f87faf72-693b-4d4d-b605-e93309fb444b" containerID="f4318790b9a476899a8f1a77d928405d8d2d6b6c2d1f1051bb67c3621a75d491" exitCode=6
Jul 16 22:25:20.074228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:20.074229    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerDied","Data":"f4318790b9a476899a8f1a77d928405d8d2d6b6c2d1f1051bb67c3621a75d491"}
Jul 16 22:25:20.074686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:20.074302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerStarted","Data":"b5dc33e816ce406d1da4acd589b0fdb8219f00433649dc2c486ca34d5dc8a0f7"}
Jul 16 22:25:20.074686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:20.074447    2555 scope.go:117] "RemoveContainer" containerID="f4318790b9a476899a8f1a77d928405d8d2d6b6c2d1f1051bb67c3621a75d491"
Jul 16 22:25:21.081636 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:21.081599    2555 generic.go:356] "Generic (PLEG): container finished" podID="f87faf72-693b-4d4d-b605-e93309fb444b" containerID="e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12" exitCode=6
Jul 16 22:25:21.082041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:21.081672    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerDied","Data":"e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12"}
Jul 16 22:25:21.082041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:21.081739    2555 scope.go:117] "RemoveContainer" containerID="f4318790b9a476899a8f1a77d928405d8d2d6b6c2d1f1051bb67c3621a75d491"
Jul 16 22:25:21.082041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:21.081918    2555 scope.go:117] "RemoveContainer" containerID="e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12"
Jul 16 22:25:21.082204 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:21.082118    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc_odh-ai-gateway-infra(f87faf72-693b-4d4d-b605-e93309fb444b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" podUID="f87faf72-693b-4d4d-b605-e93309fb444b"
Jul 16 22:25:22.088283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:22.088253    2555 scope.go:117] "RemoveContainer" containerID="e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12"
Jul 16 22:25:22.088798 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:22.088535    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc_odh-ai-gateway-infra(f87faf72-693b-4d4d-b605-e93309fb444b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" podUID="f87faf72-693b-4d4d-b605-e93309fb444b"
Jul 16 22:25:29.865420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.865384    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:29.865953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.865752    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:29.865953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.865762    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7c6a42ff-e37f-452c-9d3f-8cb4d2c3a5b2" containerName="revoke-keys"
Jul 16 22:25:29.865953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.865774    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:29.865953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.865779    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9c8c030-8aee-4eb3-8bf7-fc8acad293ef" containerName="revoke-keys"
Jul 16 22:25:29.870247 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.870230    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:29.883667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.883635    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:29.948953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:29.948914    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq\" (UID: \"9439b09f-1849-46e9-81e1-6d7fc2444400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:30.050382 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.050342    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq\" (UID: \"9439b09f-1849-46e9-81e1-6d7fc2444400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:30.050989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.050970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq\" (UID: \"9439b09f-1849-46e9-81e1-6d7fc2444400\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:30.180150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.180045    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:30.338128 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.338106    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:30.339990 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:25:30.339966    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9439b09f_1849_46e9_81e1_6d7fc2444400.slice/crio-1c9263ff52219af005a43aa71e804be1531e62fdb8aa2b98b0c5b7bf6f76fd64 WatchSource:0}: Error finding container 1c9263ff52219af005a43aa71e804be1531e62fdb8aa2b98b0c5b7bf6f76fd64: Status 404 returned error can't find the container with id 1c9263ff52219af005a43aa71e804be1531e62fdb8aa2b98b0c5b7bf6f76fd64
Jul 16 22:25:30.398406 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.398380    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:30.403201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.403183    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:30.411900 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.411878    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:30.454881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.454798    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-4qtwn\" (UID: \"6b8e7e7d-7936-4a02-bb02-10679048f67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:30.555628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.555595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-4qtwn\" (UID: \"6b8e7e7d-7936-4a02-bb02-10679048f67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:30.556141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.556124    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-4qtwn\" (UID: \"6b8e7e7d-7936-4a02-bb02-10679048f67d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:30.714579 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.714501    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:30.886306 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:30.886279    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:30.887826 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:25:30.887801    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b8e7e7d_7936_4a02_bb02_10679048f67d.slice/crio-f628343e42571a2698f3ac7a413e53f6770755aa833274a8bef38bf873ea1b09 WatchSource:0}: Error finding container f628343e42571a2698f3ac7a413e53f6770755aa833274a8bef38bf873ea1b09: Status 404 returned error can't find the container with id f628343e42571a2698f3ac7a413e53f6770755aa833274a8bef38bf873ea1b09
Jul 16 22:25:31.134271 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:31.134237    2555 generic.go:356] "Generic (PLEG): container finished" podID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerID="3decef889730893cbed841de0aa837583cca48ec9fb3520aa14600fe35f85bf9" exitCode=6
Jul 16 22:25:31.134435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:31.134305    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerDied","Data":"3decef889730893cbed841de0aa837583cca48ec9fb3520aa14600fe35f85bf9"}
Jul 16 22:25:31.134435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:31.134357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerStarted","Data":"1c9263ff52219af005a43aa71e804be1531e62fdb8aa2b98b0c5b7bf6f76fd64"}
Jul 16 22:25:31.134637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:31.134618    2555 scope.go:117] "RemoveContainer" containerID="3decef889730893cbed841de0aa837583cca48ec9fb3520aa14600fe35f85bf9"
Jul 16 22:25:31.135566 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:31.135536    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerStarted","Data":"f628343e42571a2698f3ac7a413e53f6770755aa833274a8bef38bf873ea1b09"}
Jul 16 22:25:32.143005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.142968    2555 generic.go:356] "Generic (PLEG): container finished" podID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerID="0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670" exitCode=6
Jul 16 22:25:32.143456 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.143053    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerDied","Data":"0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670"}
Jul 16 22:25:32.143456 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.143113    2555 scope.go:117] "RemoveContainer" containerID="3decef889730893cbed841de0aa837583cca48ec9fb3520aa14600fe35f85bf9"
Jul 16 22:25:32.143456 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.143387    2555 scope.go:117] "RemoveContainer" containerID="0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670"
Jul 16 22:25:32.143663 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:32.143646    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq_odh-ai-gateway-infra(9439b09f-1849-46e9-81e1-6d7fc2444400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400"
Jul 16 22:25:32.144577 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.144557    2555 generic.go:356] "Generic (PLEG): container finished" podID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerID="2f15ddb88dbf1ffcacc6609773e46437266b152db2989f42032df6b34ac7478a" exitCode=7
Jul 16 22:25:32.144669 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.144639    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerDied","Data":"2f15ddb88dbf1ffcacc6609773e46437266b152db2989f42032df6b34ac7478a"}
Jul 16 22:25:32.144790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:32.144777    2555 scope.go:117] "RemoveContainer" containerID="2f15ddb88dbf1ffcacc6609773e46437266b152db2989f42032df6b34ac7478a"
Jul 16 22:25:33.151413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.151387    2555 scope.go:117] "RemoveContainer" containerID="0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670"
Jul 16 22:25:33.151906 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:33.151666    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq_odh-ai-gateway-infra(9439b09f-1849-46e9-81e1-6d7fc2444400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400"
Jul 16 22:25:33.153116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.153093    2555 generic.go:356] "Generic (PLEG): container finished" podID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerID="616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0" exitCode=7
Jul 16 22:25:33.153216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.153152    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerDied","Data":"616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0"}
Jul 16 22:25:33.153216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.153193    2555 scope.go:117] "RemoveContainer" containerID="2f15ddb88dbf1ffcacc6609773e46437266b152db2989f42032df6b34ac7478a"
Jul 16 22:25:33.153368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.153352    2555 scope.go:117] "RemoveContainer" containerID="616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0"
Jul 16 22:25:33.153563 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:33.153544    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-4qtwn_odh-ai-gateway-infra(6b8e7e7d-7936-4a02-bb02-10679048f67d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d"
Jul 16 22:25:33.190363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:33.190337    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:25:34.159619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:34.159593    2555 scope.go:117] "RemoveContainer" containerID="616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0"
Jul 16 22:25:34.160015 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:34.159774    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-4qtwn_odh-ai-gateway-infra(6b8e7e7d-7936-4a02-bb02-10679048f67d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d"
Jul 16 22:25:34.670814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:34.670783    2555 scope.go:117] "RemoveContainer" containerID="e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12"
Jul 16 22:25:35.166446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:35.166411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerStarted","Data":"16e58ce88caed0df5eabc1a665ef50168bfce7006f14c85e642a660d6d380687"}
Jul 16 22:25:35.185610 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:35.185563    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" podStartSLOduration=17.185543973 podStartE2EDuration="17.185543973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:35.184494305 +0000 UTC m=+2190.021548103" watchObservedRunningTime="2026-07-16 22:25:35.185543973 +0000 UTC m=+2190.022597774"
Jul 16 22:25:36.174293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:36.174261    2555 generic.go:356] "Generic (PLEG): container finished" podID="f87faf72-693b-4d4d-b605-e93309fb444b" containerID="16e58ce88caed0df5eabc1a665ef50168bfce7006f14c85e642a660d6d380687" exitCode=6
Jul 16 22:25:36.174796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:36.174343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerDied","Data":"16e58ce88caed0df5eabc1a665ef50168bfce7006f14c85e642a660d6d380687"}
Jul 16 22:25:36.174796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:36.174418    2555 scope.go:117] "RemoveContainer" containerID="e221dd11a6b9889e744da5f98c462624e9b52f92684e90b951cf699a7a879d12"
Jul 16 22:25:36.174796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:36.174764    2555 scope.go:117] "RemoveContainer" containerID="16e58ce88caed0df5eabc1a665ef50168bfce7006f14c85e642a660d6d380687"
Jul 16 22:25:36.175014 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:36.174996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc_odh-ai-gateway-infra(f87faf72-693b-4d4d-b605-e93309fb444b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" podUID="f87faf72-693b-4d4d-b605-e93309fb444b"
Jul 16 22:25:36.195767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:36.195741    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:37.321911 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:37.321882    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:37.412829 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:37.412804    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca\") pod \"f87faf72-693b-4d4d-b605-e93309fb444b\" (UID: \"f87faf72-693b-4d4d-b605-e93309fb444b\") "
Jul 16 22:25:37.413188 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:37.413153    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f87faf72-693b-4d4d-b605-e93309fb444b" (UID: "f87faf72-693b-4d4d-b605-e93309fb444b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:37.513367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:37.513278    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f87faf72-693b-4d4d-b605-e93309fb444b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:25:38.188959 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:38.188925    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"
Jul 16 22:25:38.189168 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:38.188963    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc" event={"ID":"f87faf72-693b-4d4d-b605-e93309fb444b","Type":"ContainerDied","Data":"b5dc33e816ce406d1da4acd589b0fdb8219f00433649dc2c486ca34d5dc8a0f7"}
Jul 16 22:25:38.189168 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:38.189030    2555 scope.go:117] "RemoveContainer" containerID="16e58ce88caed0df5eabc1a665ef50168bfce7006f14c85e642a660d6d380687"
Jul 16 22:25:38.217018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:38.216990    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:38.220318 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:38.220296    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-gmclc"]
Jul 16 22:25:39.676584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:39.676545    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" path="/var/lib/kubelet/pods/f87faf72-693b-4d4d-b605-e93309fb444b/volumes"
Jul 16 22:25:46.670355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:46.670324    2555 scope.go:117] "RemoveContainer" containerID="616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0"
Jul 16 22:25:46.890918 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:46.890883    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:25:47.239918 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:47.239885    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerStarted","Data":"9206e16c1ce64fa2bb018541d90ab7b245f7eded7af6257d609eecda85a171c4"}
Jul 16 22:25:47.258592 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:47.258493    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" podStartSLOduration=17.258478069 podStartE2EDuration="17.258478069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:47.256498469 +0000 UTC m=+2202.093552271" watchObservedRunningTime="2026-07-16 22:25:47.258478069 +0000 UTC m=+2202.095531859"
Jul 16 22:25:47.671138 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:47.671101    2555 scope.go:117] "RemoveContainer" containerID="0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670"
Jul 16 22:25:48.247714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.247680    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerStarted","Data":"80050a17c456aab4343be50be900168474c645cbc17dd5a433e14952e728dd11"}
Jul 16 22:25:48.249500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.249453    2555 generic.go:356] "Generic (PLEG): container finished" podID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerID="9206e16c1ce64fa2bb018541d90ab7b245f7eded7af6257d609eecda85a171c4" exitCode=6
Jul 16 22:25:48.249647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.249509    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerDied","Data":"9206e16c1ce64fa2bb018541d90ab7b245f7eded7af6257d609eecda85a171c4"}
Jul 16 22:25:48.249647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.249550    2555 scope.go:117] "RemoveContainer" containerID="616e7840363dbaf9250220d9a7f83fd2ae085d3cad77e92a1fa0b6758f96d9e0"
Jul 16 22:25:48.249795 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.249778    2555 scope.go:117] "RemoveContainer" containerID="9206e16c1ce64fa2bb018541d90ab7b245f7eded7af6257d609eecda85a171c4"
Jul 16 22:25:48.250007 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:48.249987    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-4qtwn_odh-ai-gateway-infra(6b8e7e7d-7936-4a02-bb02-10679048f67d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d"
Jul 16 22:25:48.265195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.265147    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" podStartSLOduration=19.265130103 podStartE2EDuration="19.265130103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:25:48.264455965 +0000 UTC m=+2203.101509767" watchObservedRunningTime="2026-07-16 22:25:48.265130103 +0000 UTC m=+2203.102183905"
Jul 16 22:25:48.272025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.272004    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:48.548292 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548259    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:25:48.548635 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548623    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548638    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548651    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548657    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548728    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548735    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.548786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.548741    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:25:48.551827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.551808    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:25:48.558792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.558769    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:25:48.712844 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.712800    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r\" (UID: \"c2d1d657-1fac-4cad-9fe6-634d4bd24e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:25:48.813389 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.813296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r\" (UID: \"c2d1d657-1fac-4cad-9fe6-634d4bd24e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:25:48.813944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.813926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r\" (UID: \"c2d1d657-1fac-4cad-9fe6-634d4bd24e30\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:25:48.863547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.863513    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:25:48.997489 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:48.997447    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:25:48.999427 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:25:48.999391    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d1d657_1fac_4cad_9fe6_634d4bd24e30.slice/crio-8e6fbcc60c2eb3e1d834416ab179945caf751070483e3ed75dbfc554f1b10fd3 WatchSource:0}: Error finding container 8e6fbcc60c2eb3e1d834416ab179945caf751070483e3ed75dbfc554f1b10fd3: Status 404 returned error can't find the container with id 8e6fbcc60c2eb3e1d834416ab179945caf751070483e3ed75dbfc554f1b10fd3
Jul 16 22:25:49.256492 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.256435    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerStarted","Data":"8e6fbcc60c2eb3e1d834416ab179945caf751070483e3ed75dbfc554f1b10fd3"}
Jul 16 22:25:49.258246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.258214    2555 generic.go:356] "Generic (PLEG): container finished" podID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerID="80050a17c456aab4343be50be900168474c645cbc17dd5a433e14952e728dd11" exitCode=6
Jul 16 22:25:49.258362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.258285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerDied","Data":"80050a17c456aab4343be50be900168474c645cbc17dd5a433e14952e728dd11"}
Jul 16 22:25:49.258362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.258339    2555 scope.go:117] "RemoveContainer" containerID="0eab8ec24f4751f67e78fafe1ac119427975da5233e1e9cc1adcd599b1a5f670"
Jul 16 22:25:49.258763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.258736    2555 scope.go:117] "RemoveContainer" containerID="80050a17c456aab4343be50be900168474c645cbc17dd5a433e14952e728dd11"
Jul 16 22:25:49.258972 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:49.258948    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq_odh-ai-gateway-infra(9439b09f-1849-46e9-81e1-6d7fc2444400)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400"
Jul 16 22:25:49.274976 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.274946    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:49.407403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.407378    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:49.521094 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.521007    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca\") pod \"6b8e7e7d-7936-4a02-bb02-10679048f67d\" (UID: \"6b8e7e7d-7936-4a02-bb02-10679048f67d\") "
Jul 16 22:25:49.521353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.521332    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6b8e7e7d-7936-4a02-bb02-10679048f67d" (UID: "6b8e7e7d-7936-4a02-bb02-10679048f67d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:49.622076 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:49.622036    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6b8e7e7d-7936-4a02-bb02-10679048f67d-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:25:50.268531 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.268497    2555 generic.go:356] "Generic (PLEG): container finished" podID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerID="3ade7eddd8f8de84b0ab84805ef3f075da69579a0439ef9dd97f9cadc33df6ae" exitCode=7
Jul 16 22:25:50.269018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.268591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerDied","Data":"3ade7eddd8f8de84b0ab84805ef3f075da69579a0439ef9dd97f9cadc33df6ae"}
Jul 16 22:25:50.269018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.268765    2555 scope.go:117] "RemoveContainer" containerID="3ade7eddd8f8de84b0ab84805ef3f075da69579a0439ef9dd97f9cadc33df6ae"
Jul 16 22:25:50.271658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.271631    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"
Jul 16 22:25:50.271658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.271650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn" event={"ID":"6b8e7e7d-7936-4a02-bb02-10679048f67d","Type":"ContainerDied","Data":"f628343e42571a2698f3ac7a413e53f6770755aa833274a8bef38bf873ea1b09"}
Jul 16 22:25:50.271799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.271697    2555 scope.go:117] "RemoveContainer" containerID="9206e16c1ce64fa2bb018541d90ab7b245f7eded7af6257d609eecda85a171c4"
Jul 16 22:25:50.347192 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.347166    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:50.353438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.353408    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-4qtwn"]
Jul 16 22:25:50.412037 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.412015    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:50.528525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.528505    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca\") pod \"9439b09f-1849-46e9-81e1-6d7fc2444400\" (UID: \"9439b09f-1849-46e9-81e1-6d7fc2444400\") "
Jul 16 22:25:50.528817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.528797    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9439b09f-1849-46e9-81e1-6d7fc2444400" (UID: "9439b09f-1849-46e9-81e1-6d7fc2444400"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:25:50.629838 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:50.629806    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9439b09f-1849-46e9-81e1-6d7fc2444400-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:25:51.282330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.282300    2555 generic.go:356] "Generic (PLEG): container finished" podID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerID="d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca" exitCode=7
Jul 16 22:25:51.282804 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.282379    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerDied","Data":"d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca"}
Jul 16 22:25:51.282804 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.282439    2555 scope.go:117] "RemoveContainer" containerID="3ade7eddd8f8de84b0ab84805ef3f075da69579a0439ef9dd97f9cadc33df6ae"
Jul 16 22:25:51.282804 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.282686    2555 scope.go:117] "RemoveContainer" containerID="d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca"
Jul 16 22:25:51.282974 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:51.282945    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r_odh-ai-gateway-infra(c2d1d657-1fac-4cad-9fe6-634d4bd24e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30"
Jul 16 22:25:51.283687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.283616    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq" event={"ID":"9439b09f-1849-46e9-81e1-6d7fc2444400","Type":"ContainerDied","Data":"1c9263ff52219af005a43aa71e804be1531e62fdb8aa2b98b0c5b7bf6f76fd64"}
Jul 16 22:25:51.283687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.283644    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"
Jul 16 22:25:51.294787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.294750    2555 scope.go:117] "RemoveContainer" containerID="80050a17c456aab4343be50be900168474c645cbc17dd5a433e14952e728dd11"
Jul 16 22:25:51.314433 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.314394    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:51.319729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.319704    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-l57hq"]
Jul 16 22:25:51.676159 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.676085    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" path="/var/lib/kubelet/pods/6b8e7e7d-7936-4a02-bb02-10679048f67d/volumes"
Jul 16 22:25:51.676452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:51.676439    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" path="/var/lib/kubelet/pods/9439b09f-1849-46e9-81e1-6d7fc2444400/volumes"
Jul 16 22:25:52.291935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:25:52.291910    2555 scope.go:117] "RemoveContainer" containerID="d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca"
Jul 16 22:25:52.292344 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:25:52.292114    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r_odh-ai-gateway-infra(c2d1d657-1fac-4cad-9fe6-634d4bd24e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30"
Jul 16 22:26:02.671029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:02.670993    2555 scope.go:117] "RemoveContainer" containerID="d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca"
Jul 16 22:26:03.353157 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:03.353120    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerStarted","Data":"223e57b09ab8b2bcb0a7d7ff71e247241e5e56c5ebd19fed8a3134c6db3cd288"}
Jul 16 22:26:03.375985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:03.375926    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" podStartSLOduration=15.375911813 podStartE2EDuration="15.375911813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:25:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:03.371896798 +0000 UTC m=+2218.208950598" watchObservedRunningTime="2026-07-16 22:26:03.375911813 +0000 UTC m=+2218.212965613"
Jul 16 22:26:04.360350 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:04.360306    2555 generic.go:356] "Generic (PLEG): container finished" podID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerID="223e57b09ab8b2bcb0a7d7ff71e247241e5e56c5ebd19fed8a3134c6db3cd288" exitCode=7
Jul 16 22:26:04.360852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:04.360376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerDied","Data":"223e57b09ab8b2bcb0a7d7ff71e247241e5e56c5ebd19fed8a3134c6db3cd288"}
Jul 16 22:26:04.360852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:04.360449    2555 scope.go:117] "RemoveContainer" containerID="d9792633e0a74cd47171a41576595ae8a6b977554db3ee8bee54cc3d72e293ca"
Jul 16 22:26:04.360852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:04.360774    2555 scope.go:117] "RemoveContainer" containerID="223e57b09ab8b2bcb0a7d7ff71e247241e5e56c5ebd19fed8a3134c6db3cd288"
Jul 16 22:26:04.361021 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:04.361003    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r_odh-ai-gateway-infra(c2d1d657-1fac-4cad-9fe6-634d4bd24e30)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30"
Jul 16 22:26:04.385299 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:04.385267    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:26:05.501693 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:05.501668    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:26:05.561977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:05.561944    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca\") pod \"c2d1d657-1fac-4cad-9fe6-634d4bd24e30\" (UID: \"c2d1d657-1fac-4cad-9fe6-634d4bd24e30\") "
Jul 16 22:26:05.562264 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:05.562242    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c2d1d657-1fac-4cad-9fe6-634d4bd24e30" (UID: "c2d1d657-1fac-4cad-9fe6-634d4bd24e30"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:05.663019 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:05.662935    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c2d1d657-1fac-4cad-9fe6-634d4bd24e30-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:26:06.373563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:06.373534    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"
Jul 16 22:26:06.373563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:06.373559    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r" event={"ID":"c2d1d657-1fac-4cad-9fe6-634d4bd24e30","Type":"ContainerDied","Data":"8e6fbcc60c2eb3e1d834416ab179945caf751070483e3ed75dbfc554f1b10fd3"}
Jul 16 22:26:06.373817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:06.373623    2555 scope.go:117] "RemoveContainer" containerID="223e57b09ab8b2bcb0a7d7ff71e247241e5e56c5ebd19fed8a3134c6db3cd288"
Jul 16 22:26:06.392922 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:06.392896    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:26:06.397511 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:06.397487    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fq66r"]
Jul 16 22:26:07.677516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:07.677489    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" path="/var/lib/kubelet/pods/c2d1d657-1fac-4cad-9fe6-634d4bd24e30/volumes"
Jul 16 22:26:08.694944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.694915    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695232    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695243    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695255    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695260    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f87faf72-693b-4d4d-b605-e93309fb444b" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695266    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695274    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695284    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695289    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695300    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695306    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695311    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695316    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695321    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695328    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695334    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695341    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695346    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695352    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695356    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695405    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695413    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695424    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695430    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:08.695783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.695436    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:08.698399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.698383    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:08.700766 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.700747    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:26:08.700908 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.700798    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:26:08.706775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.706345    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:08.787799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.787772    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9phff\" (UID: \"a75faacf-97c1-43ea-b3a7-6302de75706f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:08.889140 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.889106    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9phff\" (UID: \"a75faacf-97c1-43ea-b3a7-6302de75706f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:08.889792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:08.889771    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-9phff\" (UID: \"a75faacf-97c1-43ea-b3a7-6302de75706f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:09.010609 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:09.010514    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:09.144229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:09.144206    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:09.146035 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:26:09.146008    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75faacf_97c1_43ea_b3a7_6302de75706f.slice/crio-37f0365fb0c9f9bd0712f18687c5024e66e347e4100817d80e8a2ea5c06dcc3e WatchSource:0}: Error finding container 37f0365fb0c9f9bd0712f18687c5024e66e347e4100817d80e8a2ea5c06dcc3e: Status 404 returned error can't find the container with id 37f0365fb0c9f9bd0712f18687c5024e66e347e4100817d80e8a2ea5c06dcc3e
Jul 16 22:26:09.391169 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:09.391133    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" event={"ID":"a75faacf-97c1-43ea-b3a7-6302de75706f","Type":"ContainerStarted","Data":"37f0365fb0c9f9bd0712f18687c5024e66e347e4100817d80e8a2ea5c06dcc3e"}
Jul 16 22:26:10.398790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:10.398753    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerID="8d41a186c2cf008fbb2c5f1b3e3c1080015477673b990bfb5aefe39ee28ea794" exitCode=6
Jul 16 22:26:10.399182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:10.398800    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" event={"ID":"a75faacf-97c1-43ea-b3a7-6302de75706f","Type":"ContainerDied","Data":"8d41a186c2cf008fbb2c5f1b3e3c1080015477673b990bfb5aefe39ee28ea794"}
Jul 16 22:26:10.399182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:10.399001    2555 scope.go:117] "RemoveContainer" containerID="8d41a186c2cf008fbb2c5f1b3e3c1080015477673b990bfb5aefe39ee28ea794"
Jul 16 22:26:11.405689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:11.405603    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerID="c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc" exitCode=6
Jul 16 22:26:11.406050 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:11.405690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" event={"ID":"a75faacf-97c1-43ea-b3a7-6302de75706f","Type":"ContainerDied","Data":"c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc"}
Jul 16 22:26:11.406050 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:11.405748    2555 scope.go:117] "RemoveContainer" containerID="8d41a186c2cf008fbb2c5f1b3e3c1080015477673b990bfb5aefe39ee28ea794"
Jul 16 22:26:11.406050 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:11.405955    2555 scope.go:117] "RemoveContainer" containerID="c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc"
Jul 16 22:26:11.406178 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:11.406158    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9phff_odh-ai-gateway-infra(a75faacf-97c1-43ea-b3a7-6302de75706f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f"
Jul 16 22:26:12.412156 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:12.412113    2555 scope.go:117] "RemoveContainer" containerID="c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc"
Jul 16 22:26:12.412645 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:12.412373    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9phff_odh-ai-gateway-infra(a75faacf-97c1-43ea-b3a7-6302de75706f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f"
Jul 16 22:26:20.459590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.459554    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:20.459998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.459960    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2d1d657-1fac-4cad-9fe6-634d4bd24e30" containerName="revoke-keys"
Jul 16 22:26:20.459998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.459975    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6b8e7e7d-7936-4a02-bb02-10679048f67d" containerName="revoke-keys"
Jul 16 22:26:20.459998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.459981    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9439b09f-1849-46e9-81e1-6d7fc2444400" containerName="revoke-keys"
Jul 16 22:26:20.464235 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.464219    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:20.482884 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.482858    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:20.590856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.590813    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tl2kv\" (UID: \"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:20.691435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.691401    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tl2kv\" (UID: \"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:20.692084 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.692064    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tl2kv\" (UID: \"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:20.774588 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.774516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:20.924315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:20.924285    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:20.925942 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:26:20.925911    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6f0691_3d8d_4a2e_93a9_6c18230bc23b.slice/crio-e334cee03999d7aa3d68b3891e6b89d8f6633ec8b312a0619859fe4913e3cfe1 WatchSource:0}: Error finding container e334cee03999d7aa3d68b3891e6b89d8f6633ec8b312a0619859fe4913e3cfe1: Status 404 returned error can't find the container with id e334cee03999d7aa3d68b3891e6b89d8f6633ec8b312a0619859fe4913e3cfe1
Jul 16 22:26:21.461529 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:21.461411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerStarted","Data":"eccd780e45e2cdefcc2183bd36b68a741661edc6ed814948a837775d5a5c26aa"}
Jul 16 22:26:21.461874 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:21.461544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerStarted","Data":"e334cee03999d7aa3d68b3891e6b89d8f6633ec8b312a0619859fe4913e3cfe1"}
Jul 16 22:26:21.480066 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:21.480013    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" podStartSLOduration=1.479996488 podStartE2EDuration="1.479996488s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:21.477531087 +0000 UTC m=+2236.314584888" watchObservedRunningTime="2026-07-16 22:26:21.479996488 +0000 UTC m=+2236.317050319"
Jul 16 22:26:22.467711 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:22.467672    2555 generic.go:356] "Generic (PLEG): container finished" podID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerID="eccd780e45e2cdefcc2183bd36b68a741661edc6ed814948a837775d5a5c26aa" exitCode=6
Jul 16 22:26:22.468097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:22.467765    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerDied","Data":"eccd780e45e2cdefcc2183bd36b68a741661edc6ed814948a837775d5a5c26aa"}
Jul 16 22:26:22.468097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:22.467936    2555 scope.go:117] "RemoveContainer" containerID="eccd780e45e2cdefcc2183bd36b68a741661edc6ed814948a837775d5a5c26aa"
Jul 16 22:26:23.474229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:23.474191    2555 generic.go:356] "Generic (PLEG): container finished" podID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerID="87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb" exitCode=6
Jul 16 22:26:23.474666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:23.474250    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerDied","Data":"87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb"}
Jul 16 22:26:23.474666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:23.474307    2555 scope.go:117] "RemoveContainer" containerID="eccd780e45e2cdefcc2183bd36b68a741661edc6ed814948a837775d5a5c26aa"
Jul 16 22:26:23.474666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:23.474494    2555 scope.go:117] "RemoveContainer" containerID="87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb"
Jul 16 22:26:23.474847 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:23.474700    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tl2kv_odh-ai-gateway-infra(4d6f0691-3d8d-4a2e-93a9-6c18230bc23b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b"
Jul 16 22:26:24.481481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:24.481448    2555 scope.go:117] "RemoveContainer" containerID="87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb"
Jul 16 22:26:24.481967 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:24.481741    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tl2kv_odh-ai-gateway-infra(4d6f0691-3d8d-4a2e-93a9-6c18230bc23b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b"
Jul 16 22:26:26.671236 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:26.671159    2555 scope.go:117] "RemoveContainer" containerID="c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc"
Jul 16 22:26:27.499602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:27.499569    2555 generic.go:356] "Generic (PLEG): container finished" podID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerID="30812ff7567512f3f5bf7f0cf2ba1fbb39c0eb7574dd582be9eba04dbaf92979" exitCode=6
Jul 16 22:26:27.499974 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:27.499937    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" event={"ID":"a75faacf-97c1-43ea-b3a7-6302de75706f","Type":"ContainerDied","Data":"30812ff7567512f3f5bf7f0cf2ba1fbb39c0eb7574dd582be9eba04dbaf92979"}
Jul 16 22:26:27.500123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:27.500021    2555 scope.go:117] "RemoveContainer" containerID="c97ca0065599c616886260ed6c53482e3ce990b08bbac56c0e73f6329b2d8ddc"
Jul 16 22:26:27.500521 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:27.500504    2555 scope.go:117] "RemoveContainer" containerID="30812ff7567512f3f5bf7f0cf2ba1fbb39c0eb7574dd582be9eba04dbaf92979"
Jul 16 22:26:27.500713 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:27.500693    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-9phff_odh-ai-gateway-infra(a75faacf-97c1-43ea-b3a7-6302de75706f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f"
Jul 16 22:26:28.557898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:28.557864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:28.694179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:28.694158    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:28.761712 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:28.761687    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca\") pod \"a75faacf-97c1-43ea-b3a7-6302de75706f\" (UID: \"a75faacf-97c1-43ea-b3a7-6302de75706f\") "
Jul 16 22:26:28.762065 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:28.762046    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a75faacf-97c1-43ea-b3a7-6302de75706f" (UID: "a75faacf-97c1-43ea-b3a7-6302de75706f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:28.863192 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:28.863099    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a75faacf-97c1-43ea-b3a7-6302de75706f-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:26:29.513878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.513848    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"
Jul 16 22:26:29.514049 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.513882    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff" event={"ID":"a75faacf-97c1-43ea-b3a7-6302de75706f","Type":"ContainerDied","Data":"37f0365fb0c9f9bd0712f18687c5024e66e347e4100817d80e8a2ea5c06dcc3e"}
Jul 16 22:26:29.514049 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.513948    2555 scope.go:117] "RemoveContainer" containerID="30812ff7567512f3f5bf7f0cf2ba1fbb39c0eb7574dd582be9eba04dbaf92979"
Jul 16 22:26:29.537007 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.536977    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:29.542951 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.542932    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-9phff"]
Jul 16 22:26:29.675805 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.675771    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" path="/var/lib/kubelet/pods/a75faacf-97c1-43ea-b3a7-6302de75706f/volumes"
Jul 16 22:26:29.917831 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.917797    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:29.918147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918135    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918148    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918157    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918163    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918178    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918183    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918247    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918255    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.918353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.918262    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a75faacf-97c1-43ea-b3a7-6302de75706f" containerName="revoke-keys"
Jul 16 22:26:29.922736 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.922715    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:29.933036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.933007    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:29.972637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:29.972609    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt\" (UID: \"d2b392ce-b380-45ac-8f05-03f19c5271bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:30.073969 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:30.073936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt\" (UID: \"d2b392ce-b380-45ac-8f05-03f19c5271bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:30.074611 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:30.074588    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt\" (UID: \"d2b392ce-b380-45ac-8f05-03f19c5271bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:30.234350 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:30.234246    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:30.390614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:30.390582    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:30.391737 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:26:30.391705    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b392ce_b380_45ac_8f05_03f19c5271bd.slice/crio-557dad6586d479b5f62b26f88ec256d8dea490f50b05dcf999ad4dfc294e0c14 WatchSource:0}: Error finding container 557dad6586d479b5f62b26f88ec256d8dea490f50b05dcf999ad4dfc294e0c14: Status 404 returned error can't find the container with id 557dad6586d479b5f62b26f88ec256d8dea490f50b05dcf999ad4dfc294e0c14
Jul 16 22:26:30.520568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:30.520482    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" event={"ID":"d2b392ce-b380-45ac-8f05-03f19c5271bd","Type":"ContainerStarted","Data":"557dad6586d479b5f62b26f88ec256d8dea490f50b05dcf999ad4dfc294e0c14"}
Jul 16 22:26:31.527127 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:31.527090    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerID="cc0ff142288d71cee903fef666f73777ff588759a8c0e62f36261cb682f437fc" exitCode=6
Jul 16 22:26:31.527593 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:31.527176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" event={"ID":"d2b392ce-b380-45ac-8f05-03f19c5271bd","Type":"ContainerDied","Data":"cc0ff142288d71cee903fef666f73777ff588759a8c0e62f36261cb682f437fc"}
Jul 16 22:26:31.527593 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:31.527324    2555 scope.go:117] "RemoveContainer" containerID="cc0ff142288d71cee903fef666f73777ff588759a8c0e62f36261cb682f437fc"
Jul 16 22:26:32.533814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:32.533775    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerID="267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe" exitCode=6
Jul 16 22:26:32.534210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:32.533859    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" event={"ID":"d2b392ce-b380-45ac-8f05-03f19c5271bd","Type":"ContainerDied","Data":"267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe"}
Jul 16 22:26:32.534210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:32.533922    2555 scope.go:117] "RemoveContainer" containerID="cc0ff142288d71cee903fef666f73777ff588759a8c0e62f36261cb682f437fc"
Jul 16 22:26:32.534210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:32.534113    2555 scope.go:117] "RemoveContainer" containerID="267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe"
Jul 16 22:26:32.534388 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:32.534368    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt_odh-ai-gateway-infra(d2b392ce-b380-45ac-8f05-03f19c5271bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd"
Jul 16 22:26:33.541073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:33.541049    2555 scope.go:117] "RemoveContainer" containerID="267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe"
Jul 16 22:26:33.541444 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:33.541262    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt_odh-ai-gateway-infra(d2b392ce-b380-45ac-8f05-03f19c5271bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd"
Jul 16 22:26:38.647076 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.647040    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:26:38.650622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.650607    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:38.670934 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.670910    2555 scope.go:117] "RemoveContainer" containerID="87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb"
Jul 16 22:26:38.671368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.671347    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:26:38.752600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.752563    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2\" (UID: \"546e36d0-43e8-451c-bd09-4821dbd6b684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:38.853616 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.853572    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2\" (UID: \"546e36d0-43e8-451c-bd09-4821dbd6b684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:38.854173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.854152    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2\" (UID: \"546e36d0-43e8-451c-bd09-4821dbd6b684\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:38.960545 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:38.960513    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:39.118909 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.118884    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:26:39.121523 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:26:39.121501    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546e36d0_43e8_451c_bd09_4821dbd6b684.slice/crio-4eae78506a52239d5495d69fb6935fde4c585f17dbc7024ce0fb094a9e92a384 WatchSource:0}: Error finding container 4eae78506a52239d5495d69fb6935fde4c585f17dbc7024ce0fb094a9e92a384: Status 404 returned error can't find the container with id 4eae78506a52239d5495d69fb6935fde4c585f17dbc7024ce0fb094a9e92a384
Jul 16 22:26:39.574824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.574785    2555 generic.go:356] "Generic (PLEG): container finished" podID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerID="f0a952f5ab675579b0d36bda0819c6cc1d37b49145d3b89c57342890a5d64bf8" exitCode=6
Jul 16 22:26:39.575025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.574862    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerDied","Data":"f0a952f5ab675579b0d36bda0819c6cc1d37b49145d3b89c57342890a5d64bf8"}
Jul 16 22:26:39.575025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.574950    2555 scope.go:117] "RemoveContainer" containerID="87a55820b5eaec9aee4cb231e42293e7a5c61b436f9da16539990973ff9444eb"
Jul 16 22:26:39.575289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.575247    2555 scope.go:117] "RemoveContainer" containerID="f0a952f5ab675579b0d36bda0819c6cc1d37b49145d3b89c57342890a5d64bf8"
Jul 16 22:26:39.575542 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:39.575515    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tl2kv_odh-ai-gateway-infra(4d6f0691-3d8d-4a2e-93a9-6c18230bc23b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b"
Jul 16 22:26:39.576816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.576785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerStarted","Data":"858634cc6c7c0b5220844a05b3a80f6dd1f030619f59191061b7e42516166942"}
Jul 16 22:26:39.576920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.576839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerStarted","Data":"4eae78506a52239d5495d69fb6935fde4c585f17dbc7024ce0fb094a9e92a384"}
Jul 16 22:26:39.625820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:39.625773    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" podStartSLOduration=1.625758982 podStartE2EDuration="1.625758982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:26:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:26:39.62293626 +0000 UTC m=+2254.459990061" watchObservedRunningTime="2026-07-16 22:26:39.625758982 +0000 UTC m=+2254.462812791"
Jul 16 22:26:40.585369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.585329    2555 generic.go:356] "Generic (PLEG): container finished" podID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerID="858634cc6c7c0b5220844a05b3a80f6dd1f030619f59191061b7e42516166942" exitCode=7
Jul 16 22:26:40.585775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.585385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerDied","Data":"858634cc6c7c0b5220844a05b3a80f6dd1f030619f59191061b7e42516166942"}
Jul 16 22:26:40.585775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.585684    2555 scope.go:117] "RemoveContainer" containerID="858634cc6c7c0b5220844a05b3a80f6dd1f030619f59191061b7e42516166942"
Jul 16 22:26:40.614925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.614897    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:40.753407 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.753385    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:40.871418 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.871336    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca\") pod \"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b\" (UID: \"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b\") "
Jul 16 22:26:40.871732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.871707    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" (UID: "4d6f0691-3d8d-4a2e-93a9-6c18230bc23b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:40.972873 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:40.972843    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:26:41.592362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.592326    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"
Jul 16 22:26:41.592362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.592349    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv" event={"ID":"4d6f0691-3d8d-4a2e-93a9-6c18230bc23b","Type":"ContainerDied","Data":"e334cee03999d7aa3d68b3891e6b89d8f6633ec8b312a0619859fe4913e3cfe1"}
Jul 16 22:26:41.592876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.592419    2555 scope.go:117] "RemoveContainer" containerID="f0a952f5ab675579b0d36bda0819c6cc1d37b49145d3b89c57342890a5d64bf8"
Jul 16 22:26:41.594416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.594388    2555 generic.go:356] "Generic (PLEG): container finished" podID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerID="765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506" exitCode=7
Jul 16 22:26:41.594581 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.594491    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerDied","Data":"765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506"}
Jul 16 22:26:41.594661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.594638    2555 scope.go:117] "RemoveContainer" containerID="765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506"
Jul 16 22:26:41.594835 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:41.594816    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2_odh-ai-gateway-infra(546e36d0-43e8-451c-bd09-4821dbd6b684)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684"
Jul 16 22:26:41.601514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.601480    2555 scope.go:117] "RemoveContainer" containerID="858634cc6c7c0b5220844a05b3a80f6dd1f030619f59191061b7e42516166942"
Jul 16 22:26:41.628981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.628947    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:41.636698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.636673    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tl2kv"]
Jul 16 22:26:41.676943 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:41.676868    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" path="/var/lib/kubelet/pods/4d6f0691-3d8d-4a2e-93a9-6c18230bc23b/volumes"
Jul 16 22:26:42.602488 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:42.602436    2555 scope.go:117] "RemoveContainer" containerID="765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506"
Jul 16 22:26:42.602873 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:42.602659    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2_odh-ai-gateway-infra(546e36d0-43e8-451c-bd09-4821dbd6b684)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684"
Jul 16 22:26:48.670330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:48.670301    2555 scope.go:117] "RemoveContainer" containerID="267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe"
Jul 16 22:26:49.642057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:49.641968    2555 generic.go:356] "Generic (PLEG): container finished" podID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerID="0a53949a3970b0d7312c77262e0dc4201d5147aaee173afa0a9c6572a29d53ed" exitCode=6
Jul 16 22:26:49.642057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:49.642041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" event={"ID":"d2b392ce-b380-45ac-8f05-03f19c5271bd","Type":"ContainerDied","Data":"0a53949a3970b0d7312c77262e0dc4201d5147aaee173afa0a9c6572a29d53ed"}
Jul 16 22:26:49.642251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:49.642098    2555 scope.go:117] "RemoveContainer" containerID="267b2e806abe4135a45f8ceb919ab5ae4a3808b1561e13e4163e7c9535f8bbfe"
Jul 16 22:26:49.642479 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:49.642445    2555 scope.go:117] "RemoveContainer" containerID="0a53949a3970b0d7312c77262e0dc4201d5147aaee173afa0a9c6572a29d53ed"
Jul 16 22:26:49.642668 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:49.642645    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt_odh-ai-gateway-infra(d2b392ce-b380-45ac-8f05-03f19c5271bd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd"
Jul 16 22:26:50.676451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:50.676417    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:50.811047 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:50.811026    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:50.964302 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:50.964216    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca\") pod \"d2b392ce-b380-45ac-8f05-03f19c5271bd\" (UID: \"d2b392ce-b380-45ac-8f05-03f19c5271bd\") "
Jul 16 22:26:50.964591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:50.964569    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d2b392ce-b380-45ac-8f05-03f19c5271bd" (UID: "d2b392ce-b380-45ac-8f05-03f19c5271bd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:26:51.065033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.065004    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d2b392ce-b380-45ac-8f05-03f19c5271bd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:26:51.655659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.655633    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"
Jul 16 22:26:51.655808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.655625    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt" event={"ID":"d2b392ce-b380-45ac-8f05-03f19c5271bd","Type":"ContainerDied","Data":"557dad6586d479b5f62b26f88ec256d8dea490f50b05dcf999ad4dfc294e0c14"}
Jul 16 22:26:51.655808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.655772    2555 scope.go:117] "RemoveContainer" containerID="0a53949a3970b0d7312c77262e0dc4201d5147aaee173afa0a9c6572a29d53ed"
Jul 16 22:26:51.682944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.682915    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:51.698164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:51.698135    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-b8bdt"]
Jul 16 22:26:53.675875 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:53.675844    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" path="/var/lib/kubelet/pods/d2b392ce-b380-45ac-8f05-03f19c5271bd/volumes"
Jul 16 22:26:57.670561 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:57.670531    2555 scope.go:117] "RemoveContainer" containerID="765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506"
Jul 16 22:26:58.694413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:58.694329    2555 generic.go:356] "Generic (PLEG): container finished" podID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerID="2659b7282ed204a6a9c403d955b3cc0e62d9f8de3b53977b1478ccc0a573161f" exitCode=7
Jul 16 22:26:58.694413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:58.694402    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerDied","Data":"2659b7282ed204a6a9c403d955b3cc0e62d9f8de3b53977b1478ccc0a573161f"}
Jul 16 22:26:58.694878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:58.694481    2555 scope.go:117] "RemoveContainer" containerID="765090264ada17fbca66da4d94b796036f5c8dac15b63c734a987583c4318506"
Jul 16 22:26:58.694878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:58.694731    2555 scope.go:117] "RemoveContainer" containerID="2659b7282ed204a6a9c403d955b3cc0e62d9f8de3b53977b1478ccc0a573161f"
Jul 16 22:26:58.694964 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:26:58.694943    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2_odh-ai-gateway-infra(546e36d0-43e8-451c-bd09-4821dbd6b684)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684"
Jul 16 22:26:59.722174 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:59.722142    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:26:59.861064 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:59.861044    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:26:59.938143 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:59.938117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca\") pod \"546e36d0-43e8-451c-bd09-4821dbd6b684\" (UID: \"546e36d0-43e8-451c-bd09-4821dbd6b684\") "
Jul 16 22:26:59.938446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:26:59.938426    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "546e36d0-43e8-451c-bd09-4821dbd6b684" (UID: "546e36d0-43e8-451c-bd09-4821dbd6b684"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:00.038981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.038947    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/546e36d0-43e8-451c-bd09-4821dbd6b684-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:27:00.707457 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.707428    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"
Jul 16 22:27:00.707647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.707484    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2" event={"ID":"546e36d0-43e8-451c-bd09-4821dbd6b684","Type":"ContainerDied","Data":"4eae78506a52239d5495d69fb6935fde4c585f17dbc7024ce0fb094a9e92a384"}
Jul 16 22:27:00.707647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.707539    2555 scope.go:117] "RemoveContainer" containerID="2659b7282ed204a6a9c403d955b3cc0e62d9f8de3b53977b1478ccc0a573161f"
Jul 16 22:27:00.734020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.733989    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:27:00.737250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:00.737216    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-rzzz2"]
Jul 16 22:27:01.676151 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:01.676112    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" path="/var/lib/kubelet/pods/546e36d0-43e8-451c-bd09-4821dbd6b684/volumes"
Jul 16 22:27:08.775034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.774998    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:08.775519 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775500    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.775519 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775518    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775533    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775541    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775554    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775562    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775574    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775582    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775606    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775613    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775629    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.775661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775637    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775725    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775736    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775745    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775754    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775765    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775773    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775781    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:08.776139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.775790    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:08.778922 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.778904    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:08.781477 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.781434    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:27:08.781611 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.781512    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:27:08.787117 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.787087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:08.816553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.816512    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k\" (UID: \"c3542b2e-d2be-45db-aebe-d4f602712b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:08.917001 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.916963    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k\" (UID: \"c3542b2e-d2be-45db-aebe-d4f602712b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:08.917673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:08.917655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k\" (UID: \"c3542b2e-d2be-45db-aebe-d4f602712b62\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:09.091150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.091111    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:09.225988 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.225956    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:09.227804 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:27:09.227775    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3542b2e_d2be_45db_aebe_d4f602712b62.slice/crio-65e759bf08b0e848d472eedccefda1faa620500fb3cdebb7de5cb84e64a2d75e WatchSource:0}: Error finding container 65e759bf08b0e848d472eedccefda1faa620500fb3cdebb7de5cb84e64a2d75e: Status 404 returned error can't find the container with id 65e759bf08b0e848d472eedccefda1faa620500fb3cdebb7de5cb84e64a2d75e
Jul 16 22:27:09.755521 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.755489    2555 generic.go:356] "Generic (PLEG): container finished" podID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerID="b4f747b8dfde3d75330de6350a728f4cd0a625179dd063bbb0aa664ae858f0d3" exitCode=6
Jul 16 22:27:09.755726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.755569    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" event={"ID":"c3542b2e-d2be-45db-aebe-d4f602712b62","Type":"ContainerDied","Data":"b4f747b8dfde3d75330de6350a728f4cd0a625179dd063bbb0aa664ae858f0d3"}
Jul 16 22:27:09.755726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.755626    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" event={"ID":"c3542b2e-d2be-45db-aebe-d4f602712b62","Type":"ContainerStarted","Data":"65e759bf08b0e848d472eedccefda1faa620500fb3cdebb7de5cb84e64a2d75e"}
Jul 16 22:27:09.755807 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:09.755726    2555 scope.go:117] "RemoveContainer" containerID="b4f747b8dfde3d75330de6350a728f4cd0a625179dd063bbb0aa664ae858f0d3"
Jul 16 22:27:10.763698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:10.763610    2555 generic.go:356] "Generic (PLEG): container finished" podID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerID="1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1" exitCode=6
Jul 16 22:27:10.764075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:10.763689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" event={"ID":"c3542b2e-d2be-45db-aebe-d4f602712b62","Type":"ContainerDied","Data":"1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1"}
Jul 16 22:27:10.764075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:10.763745    2555 scope.go:117] "RemoveContainer" containerID="b4f747b8dfde3d75330de6350a728f4cd0a625179dd063bbb0aa664ae858f0d3"
Jul 16 22:27:10.764075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:10.763916    2555 scope.go:117] "RemoveContainer" containerID="1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1"
Jul 16 22:27:10.764257 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:10.764141    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k_odh-ai-gateway-infra(c3542b2e-d2be-45db-aebe-d4f602712b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62"
Jul 16 22:27:11.771015 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:11.770981    2555 scope.go:117] "RemoveContainer" containerID="1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1"
Jul 16 22:27:11.771407 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:11.771231    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k_odh-ai-gateway-infra(c3542b2e-d2be-45db-aebe-d4f602712b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62"
Jul 16 22:27:20.519216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519178    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519559    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519571    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519590    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519596    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d2b392ce-b380-45ac-8f05-03f19c5271bd" containerName="revoke-keys"
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519605    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:20.519615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519612    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="546e36d0-43e8-451c-bd09-4821dbd6b684" containerName="revoke-keys"
Jul 16 22:27:20.519811 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.519677    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d6f0691-3d8d-4a2e-93a9-6c18230bc23b" containerName="revoke-keys"
Jul 16 22:27:20.524076 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.524060    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:20.531961 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.531402    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:20.617882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.617846    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5gxz6\" (UID: \"08d6148e-5a1f-487d-a6af-0b3dc3b4704b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:20.719334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.719296    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5gxz6\" (UID: \"08d6148e-5a1f-487d-a6af-0b3dc3b4704b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:20.719977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.719958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5gxz6\" (UID: \"08d6148e-5a1f-487d-a6af-0b3dc3b4704b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:20.837051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.837025    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:20.970285 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:20.970260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:20.971921 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:27:20.971896    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08d6148e_5a1f_487d_a6af_0b3dc3b4704b.slice/crio-3fdaaf5754034bfc0eb2f1f50653b32d19f095985c75b42f01055ee0b1dbeed3 WatchSource:0}: Error finding container 3fdaaf5754034bfc0eb2f1f50653b32d19f095985c75b42f01055ee0b1dbeed3: Status 404 returned error can't find the container with id 3fdaaf5754034bfc0eb2f1f50653b32d19f095985c75b42f01055ee0b1dbeed3
Jul 16 22:27:21.825319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:21.825282    2555 generic.go:356] "Generic (PLEG): container finished" podID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerID="c60a7f65826e169d869010d158f93d4aa42965e81cb6f8993fa2f798f1b857c1" exitCode=6
Jul 16 22:27:21.825722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:21.825375    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerDied","Data":"c60a7f65826e169d869010d158f93d4aa42965e81cb6f8993fa2f798f1b857c1"}
Jul 16 22:27:21.825722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:21.825430    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerStarted","Data":"3fdaaf5754034bfc0eb2f1f50653b32d19f095985c75b42f01055ee0b1dbeed3"}
Jul 16 22:27:21.825722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:21.825577    2555 scope.go:117] "RemoveContainer" containerID="c60a7f65826e169d869010d158f93d4aa42965e81cb6f8993fa2f798f1b857c1"
Jul 16 22:27:22.832696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:22.832661    2555 generic.go:356] "Generic (PLEG): container finished" podID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerID="1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd" exitCode=6
Jul 16 22:27:22.833116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:22.832743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerDied","Data":"1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd"}
Jul 16 22:27:22.833116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:22.832803    2555 scope.go:117] "RemoveContainer" containerID="c60a7f65826e169d869010d158f93d4aa42965e81cb6f8993fa2f798f1b857c1"
Jul 16 22:27:22.833116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:22.833009    2555 scope.go:117] "RemoveContainer" containerID="1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd"
Jul 16 22:27:22.833255 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:22.833212    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5gxz6_odh-ai-gateway-infra(08d6148e-5a1f-487d-a6af-0b3dc3b4704b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b"
Jul 16 22:27:23.670763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:23.670726    2555 scope.go:117] "RemoveContainer" containerID="1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1"
Jul 16 22:27:23.840717 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:23.840684    2555 scope.go:117] "RemoveContainer" containerID="1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd"
Jul 16 22:27:23.841106 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:23.840873    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5gxz6_odh-ai-gateway-infra(08d6148e-5a1f-487d-a6af-0b3dc3b4704b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b"
Jul 16 22:27:24.849021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:24.848990    2555 generic.go:356] "Generic (PLEG): container finished" podID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerID="f432359dc7974134b192c1e68b76c0109e374f7965d6bc643d5e3f64e0be9d4d" exitCode=6
Jul 16 22:27:24.849021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:24.849027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" event={"ID":"c3542b2e-d2be-45db-aebe-d4f602712b62","Type":"ContainerDied","Data":"f432359dc7974134b192c1e68b76c0109e374f7965d6bc643d5e3f64e0be9d4d"}
Jul 16 22:27:24.849445 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:24.849072    2555 scope.go:117] "RemoveContainer" containerID="1567b16aceda11a7eb6f14a9b3821cc8289284b6186c91a5719d06b52fd9aeb1"
Jul 16 22:27:24.849445 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:24.849357    2555 scope.go:117] "RemoveContainer" containerID="f432359dc7974134b192c1e68b76c0109e374f7965d6bc643d5e3f64e0be9d4d"
Jul 16 22:27:24.849612 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:24.849593    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k_odh-ai-gateway-infra(c3542b2e-d2be-45db-aebe-d4f602712b62)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62"
Jul 16 22:27:25.874145 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:25.874115    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:26.000214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.000188    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:26.073751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.073718    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca\") pod \"c3542b2e-d2be-45db-aebe-d4f602712b62\" (UID: \"c3542b2e-d2be-45db-aebe-d4f602712b62\") "
Jul 16 22:27:26.074070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.074048    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c3542b2e-d2be-45db-aebe-d4f602712b62" (UID: "c3542b2e-d2be-45db-aebe-d4f602712b62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:26.175376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.175286    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c3542b2e-d2be-45db-aebe-d4f602712b62-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:27:26.862416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.862386    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"
Jul 16 22:27:26.862647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.862383    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k" event={"ID":"c3542b2e-d2be-45db-aebe-d4f602712b62","Type":"ContainerDied","Data":"65e759bf08b0e848d472eedccefda1faa620500fb3cdebb7de5cb84e64a2d75e"}
Jul 16 22:27:26.862647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.862530    2555 scope.go:117] "RemoveContainer" containerID="f432359dc7974134b192c1e68b76c0109e374f7965d6bc643d5e3f64e0be9d4d"
Jul 16 22:27:26.889376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.889342    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:26.896440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:26.896413    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-87m6k"]
Jul 16 22:27:27.676027 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:27.675990    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" path="/var/lib/kubelet/pods/c3542b2e-d2be-45db-aebe-d4f602712b62/volumes"
Jul 16 22:27:29.961819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.961779    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:29.962198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962167    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962178    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962191    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962197    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962227    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962234    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962282    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962296    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.962329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.962306    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c3542b2e-d2be-45db-aebe-d4f602712b62" containerName="revoke-keys"
Jul 16 22:27:29.966820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.966804    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:29.973228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:29.973204    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:30.109173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.109135    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw\" (UID: \"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:30.210499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.210445    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw\" (UID: \"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:30.211107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.211080    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw\" (UID: \"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:30.278133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.278105    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:30.430853 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.430817    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:30.431310 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:27:30.431285    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1ddb245_8a5d_4fb8_8f26_3864c5e8ad37.slice/crio-257e50e8b6ac03d798bef134e4de2d2d34007676c2559105b55b23edb28c1c77 WatchSource:0}: Error finding container 257e50e8b6ac03d798bef134e4de2d2d34007676c2559105b55b23edb28c1c77: Status 404 returned error can't find the container with id 257e50e8b6ac03d798bef134e4de2d2d34007676c2559105b55b23edb28c1c77
Jul 16 22:27:30.885457 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.885368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerStarted","Data":"6ee8c3412a5770c3e182026a25e8cf1b8c1780c3a1088c8758e3101c12d6ad0d"}
Jul 16 22:27:30.885457 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.885416    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerStarted","Data":"257e50e8b6ac03d798bef134e4de2d2d34007676c2559105b55b23edb28c1c77"}
Jul 16 22:27:30.927950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:30.927904    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" podStartSLOduration=1.927889393 podStartE2EDuration="1.927889393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:30.925603036 +0000 UTC m=+2305.762656836" watchObservedRunningTime="2026-07-16 22:27:30.927889393 +0000 UTC m=+2305.764943215"
Jul 16 22:27:31.892109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:31.892071    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerID="6ee8c3412a5770c3e182026a25e8cf1b8c1780c3a1088c8758e3101c12d6ad0d" exitCode=6
Jul 16 22:27:31.892526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:31.892155    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerDied","Data":"6ee8c3412a5770c3e182026a25e8cf1b8c1780c3a1088c8758e3101c12d6ad0d"}
Jul 16 22:27:31.892526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:31.892342    2555 scope.go:117] "RemoveContainer" containerID="6ee8c3412a5770c3e182026a25e8cf1b8c1780c3a1088c8758e3101c12d6ad0d"
Jul 16 22:27:32.904977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:32.904941    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerID="908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29" exitCode=6
Jul 16 22:27:32.905417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:32.905017    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerDied","Data":"908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29"}
Jul 16 22:27:32.905417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:32.905096    2555 scope.go:117] "RemoveContainer" containerID="6ee8c3412a5770c3e182026a25e8cf1b8c1780c3a1088c8758e3101c12d6ad0d"
Jul 16 22:27:32.905417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:32.905262    2555 scope.go:117] "RemoveContainer" containerID="908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29"
Jul 16 22:27:32.905554 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:32.905513    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw_odh-ai-gateway-infra(f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37"
Jul 16 22:27:33.912041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:33.912009    2555 scope.go:117] "RemoveContainer" containerID="908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29"
Jul 16 22:27:33.912421 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:33.912213    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw_odh-ai-gateway-infra(f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37"
Jul 16 22:27:37.671169 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:37.671140    2555 scope.go:117] "RemoveContainer" containerID="1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd"
Jul 16 22:27:37.936503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:37.936418    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerStarted","Data":"dfb061bc4afa7017535ebbdacbfd96caf153f7a532d6a1c9e2e871db868b6539"}
Jul 16 22:27:37.961385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:37.961328    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" podStartSLOduration=17.961312292 podStartE2EDuration="17.961312292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:27:37.95765195 +0000 UTC m=+2312.794705751" watchObservedRunningTime="2026-07-16 22:27:37.961312292 +0000 UTC m=+2312.798366091"
Jul 16 22:27:38.654957 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.654868    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:38.658366 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.658347    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:38.668216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.668170    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:38.786393 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.786358    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m\" (UID: \"9598449c-193b-478f-b2f3-b1a8bd5f29da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:38.886827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.886793    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m\" (UID: \"9598449c-193b-478f-b2f3-b1a8bd5f29da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:38.887415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.887382    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m\" (UID: \"9598449c-193b-478f-b2f3-b1a8bd5f29da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:38.943765 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.943682    2555 generic.go:356] "Generic (PLEG): container finished" podID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerID="dfb061bc4afa7017535ebbdacbfd96caf153f7a532d6a1c9e2e871db868b6539" exitCode=6
Jul 16 22:27:38.943765 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.943727    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerDied","Data":"dfb061bc4afa7017535ebbdacbfd96caf153f7a532d6a1c9e2e871db868b6539"}
Jul 16 22:27:38.943972 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.943774    2555 scope.go:117] "RemoveContainer" containerID="1a01d5b86214194a16066cf16f786a29e151287eb002ccad29c3545f37d12dbd"
Jul 16 22:27:38.944101 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.944069    2555 scope.go:117] "RemoveContainer" containerID="dfb061bc4afa7017535ebbdacbfd96caf153f7a532d6a1c9e2e871db868b6539"
Jul 16 22:27:38.944314 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:38.944289    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5gxz6_odh-ai-gateway-infra(08d6148e-5a1f-487d-a6af-0b3dc3b4704b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b"
Jul 16 22:27:38.970279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.970243    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:38.970422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:38.970313    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:39.108221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:39.108186    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:39.110088 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:27:39.110055    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9598449c_193b_478f_b2f3_b1a8bd5f29da.slice/crio-2141562fe15a5ac69c49f446d31e649b9de754430edac5c0b7e738ba5beecdbd WatchSource:0}: Error finding container 2141562fe15a5ac69c49f446d31e649b9de754430edac5c0b7e738ba5beecdbd: Status 404 returned error can't find the container with id 2141562fe15a5ac69c49f446d31e649b9de754430edac5c0b7e738ba5beecdbd
Jul 16 22:27:39.950949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:39.950912    2555 generic.go:356] "Generic (PLEG): container finished" podID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerID="56d3157bf417dfb0aa74414a3a6ea5bcc46c9f7770cfa2dc7376889826567de3" exitCode=7
Jul 16 22:27:39.951397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:39.950998    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerDied","Data":"56d3157bf417dfb0aa74414a3a6ea5bcc46c9f7770cfa2dc7376889826567de3"}
Jul 16 22:27:39.951397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:39.951071    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerStarted","Data":"2141562fe15a5ac69c49f446d31e649b9de754430edac5c0b7e738ba5beecdbd"}
Jul 16 22:27:39.951397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:39.951187    2555 scope.go:117] "RemoveContainer" containerID="56d3157bf417dfb0aa74414a3a6ea5bcc46c9f7770cfa2dc7376889826567de3"
Jul 16 22:27:40.101393 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.101368    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:40.197768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.197724    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca\") pod \"08d6148e-5a1f-487d-a6af-0b3dc3b4704b\" (UID: \"08d6148e-5a1f-487d-a6af-0b3dc3b4704b\") "
Jul 16 22:27:40.198075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.198053    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "08d6148e-5a1f-487d-a6af-0b3dc3b4704b" (UID: "08d6148e-5a1f-487d-a6af-0b3dc3b4704b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:40.298951 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.298914    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/08d6148e-5a1f-487d-a6af-0b3dc3b4704b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:27:40.961308 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.961272    2555 generic.go:356] "Generic (PLEG): container finished" podID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerID="13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f" exitCode=7
Jul 16 22:27:40.961780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.961363    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerDied","Data":"13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f"}
Jul 16 22:27:40.961780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.961431    2555 scope.go:117] "RemoveContainer" containerID="56d3157bf417dfb0aa74414a3a6ea5bcc46c9f7770cfa2dc7376889826567de3"
Jul 16 22:27:40.961780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.961667    2555 scope.go:117] "RemoveContainer" containerID="13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f"
Jul 16 22:27:40.961973 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:40.961893    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m_odh-ai-gateway-infra(9598449c-193b-478f-b2f3-b1a8bd5f29da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da"
Jul 16 22:27:40.962865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.962846    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"
Jul 16 22:27:40.962979 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.962876    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6" event={"ID":"08d6148e-5a1f-487d-a6af-0b3dc3b4704b","Type":"ContainerDied","Data":"3fdaaf5754034bfc0eb2f1f50653b32d19f095985c75b42f01055ee0b1dbeed3"}
Jul 16 22:27:40.973063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.973040    2555 scope.go:117] "RemoveContainer" containerID="dfb061bc4afa7017535ebbdacbfd96caf153f7a532d6a1c9e2e871db868b6539"
Jul 16 22:27:40.990999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.990974    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:40.994935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:40.994908    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5gxz6"]
Jul 16 22:27:41.676502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:41.676446    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" path="/var/lib/kubelet/pods/08d6148e-5a1f-487d-a6af-0b3dc3b4704b/volumes"
Jul 16 22:27:41.970131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:41.970046    2555 scope.go:117] "RemoveContainer" containerID="13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f"
Jul 16 22:27:41.970602 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:41.970300    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m_odh-ai-gateway-infra(9598449c-193b-478f-b2f3-b1a8bd5f29da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da"
Jul 16 22:27:47.670539 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:47.670500    2555 scope.go:117] "RemoveContainer" containerID="908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29"
Jul 16 22:27:49.010892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:49.010860    2555 generic.go:356] "Generic (PLEG): container finished" podID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerID="73cb08fce1e8df3153e53d0a2a3467b11b4a22efcca6de2f679bc8df994d1183" exitCode=6
Jul 16 22:27:49.010892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:49.010899    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerDied","Data":"73cb08fce1e8df3153e53d0a2a3467b11b4a22efcca6de2f679bc8df994d1183"}
Jul 16 22:27:49.011403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:49.010947    2555 scope.go:117] "RemoveContainer" containerID="908e38bb66573609067392292254b97138e3b2509bf70d8b6af78c6cf072cc29"
Jul 16 22:27:49.011403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:49.011270    2555 scope.go:117] "RemoveContainer" containerID="73cb08fce1e8df3153e53d0a2a3467b11b4a22efcca6de2f679bc8df994d1183"
Jul 16 22:27:49.011551 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:49.011530    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw_odh-ai-gateway-infra(f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37"
Jul 16 22:27:50.040454 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:50.040416    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:50.182016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:50.181988    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:50.274255 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:50.274201    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca\") pod \"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37\" (UID: \"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37\") "
Jul 16 22:27:50.274619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:50.274599    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" (UID: "f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:50.375608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:50.375525    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:27:51.024295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.024264    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"
Jul 16 22:27:51.024295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.024281    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw" event={"ID":"f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37","Type":"ContainerDied","Data":"257e50e8b6ac03d798bef134e4de2d2d34007676c2559105b55b23edb28c1c77"}
Jul 16 22:27:51.024584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.024353    2555 scope.go:117] "RemoveContainer" containerID="73cb08fce1e8df3153e53d0a2a3467b11b4a22efcca6de2f679bc8df994d1183"
Jul 16 22:27:51.047851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.047826    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:51.053371 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.053347    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-ck6lw"]
Jul 16 22:27:51.676668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:51.676627    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" path="/var/lib/kubelet/pods/f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37/volumes"
Jul 16 22:27:55.676786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:55.676747    2555 scope.go:117] "RemoveContainer" containerID="13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f"
Jul 16 22:27:56.052484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:56.052428    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerStarted","Data":"61ae7b10c0cde4b5adaf21a57c9073c8234529343fc43c4c2ece09f4840fa3e8"}
Jul 16 22:27:57.059361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:57.059330    2555 generic.go:356] "Generic (PLEG): container finished" podID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerID="61ae7b10c0cde4b5adaf21a57c9073c8234529343fc43c4c2ece09f4840fa3e8" exitCode=7
Jul 16 22:27:57.059825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:57.059380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerDied","Data":"61ae7b10c0cde4b5adaf21a57c9073c8234529343fc43c4c2ece09f4840fa3e8"}
Jul 16 22:27:57.059825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:57.059487    2555 scope.go:117] "RemoveContainer" containerID="13cb5740f13030423c1d899a9f990bab66b5ba0f97f53481c1e72bc52604ee7f"
Jul 16 22:27:57.059825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:57.059753    2555 scope.go:117] "RemoveContainer" containerID="61ae7b10c0cde4b5adaf21a57c9073c8234529343fc43c4c2ece09f4840fa3e8"
Jul 16 22:27:57.060008 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:27:57.059985    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m_odh-ai-gateway-infra(9598449c-193b-478f-b2f3-b1a8bd5f29da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da"
Jul 16 22:27:57.100889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:57.100847    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:58.200517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.200491    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:58.238436 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.238405    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca\") pod \"9598449c-193b-478f-b2f3-b1a8bd5f29da\" (UID: \"9598449c-193b-478f-b2f3-b1a8bd5f29da\") "
Jul 16 22:27:58.238778 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.238755    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9598449c-193b-478f-b2f3-b1a8bd5f29da" (UID: "9598449c-193b-478f-b2f3-b1a8bd5f29da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:27:58.339505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.339396    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9598449c-193b-478f-b2f3-b1a8bd5f29da-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:27:58.827090 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827057    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:27:58.827398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827385    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827399    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827415    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827422    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827426    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827433    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827438    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827444    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827450    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827456    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827474    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827503    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827510    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827580    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827587    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827592    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827599    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827604    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:27:58.827839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.827608    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:27:58.830549 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.830527    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:27:58.840722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.840700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:27:58.844212 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.844189    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55\" (UID: \"359ed37e-d534-47dd-a1b7-66b553bc2efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:27:58.944724 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.944696    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55\" (UID: \"359ed37e-d534-47dd-a1b7-66b553bc2efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:27:58.945266 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:58.945248    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55\" (UID: \"359ed37e-d534-47dd-a1b7-66b553bc2efc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:27:59.073265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.073227    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m" event={"ID":"9598449c-193b-478f-b2f3-b1a8bd5f29da","Type":"ContainerDied","Data":"2141562fe15a5ac69c49f446d31e649b9de754430edac5c0b7e738ba5beecdbd"}
Jul 16 22:27:59.073265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.073246    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"
Jul 16 22:27:59.073560 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.073297    2555 scope.go:117] "RemoveContainer" containerID="61ae7b10c0cde4b5adaf21a57c9073c8234529343fc43c4c2ece09f4840fa3e8"
Jul 16 22:27:59.096619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.096592    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:59.102299 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.102277    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-ltn8m"]
Jul 16 22:27:59.141283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.141245    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:27:59.302059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.302034    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:27:59.304100 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:27:59.304069    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod359ed37e_d534_47dd_a1b7_66b553bc2efc.slice/crio-3c0a5b052e03c863dd80c3d9fb7ee675365b5eef09329b37a427c86d1a98008e WatchSource:0}: Error finding container 3c0a5b052e03c863dd80c3d9fb7ee675365b5eef09329b37a427c86d1a98008e: Status 404 returned error can't find the container with id 3c0a5b052e03c863dd80c3d9fb7ee675365b5eef09329b37a427c86d1a98008e
Jul 16 22:27:59.676175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:27:59.676143    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" path="/var/lib/kubelet/pods/9598449c-193b-478f-b2f3-b1a8bd5f29da/volumes"
Jul 16 22:28:00.081339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:00.081305    2555 generic.go:356] "Generic (PLEG): container finished" podID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerID="5c38850ef05f524d6a27cd41b3d241c9ebaca50df61c751b50e010a941c69f5b" exitCode=6
Jul 16 22:28:00.081512 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:00.081347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerDied","Data":"5c38850ef05f524d6a27cd41b3d241c9ebaca50df61c751b50e010a941c69f5b"}
Jul 16 22:28:00.081512 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:00.081390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerStarted","Data":"3c0a5b052e03c863dd80c3d9fb7ee675365b5eef09329b37a427c86d1a98008e"}
Jul 16 22:28:00.081639 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:00.081570    2555 scope.go:117] "RemoveContainer" containerID="5c38850ef05f524d6a27cd41b3d241c9ebaca50df61c751b50e010a941c69f5b"
Jul 16 22:28:01.088208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:01.088162    2555 generic.go:356] "Generic (PLEG): container finished" podID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerID="ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c" exitCode=6
Jul 16 22:28:01.088621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:01.088247    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerDied","Data":"ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c"}
Jul 16 22:28:01.088621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:01.088327    2555 scope.go:117] "RemoveContainer" containerID="5c38850ef05f524d6a27cd41b3d241c9ebaca50df61c751b50e010a941c69f5b"
Jul 16 22:28:01.088621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:01.088536    2555 scope.go:117] "RemoveContainer" containerID="ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c"
Jul 16 22:28:01.088788 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:01.088766    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55_odh-ai-gateway-infra(359ed37e-d534-47dd-a1b7-66b553bc2efc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc"
Jul 16 22:28:02.095191 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:02.095155    2555 scope.go:117] "RemoveContainer" containerID="ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c"
Jul 16 22:28:02.095613 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:02.095382    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55_odh-ai-gateway-infra(359ed37e-d534-47dd-a1b7-66b553bc2efc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc"
Jul 16 22:28:15.674125 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:15.674092    2555 scope.go:117] "RemoveContainer" containerID="ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c"
Jul 16 22:28:16.166425 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:16.166390    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerStarted","Data":"c5dcacf91ad79f09dfbd54f873767098677f88d2a06557859b8d849ea406e612"}
Jul 16 22:28:16.185426 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:16.185328    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" podStartSLOduration=18.185314714 podStartE2EDuration="18.185314714s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:27:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:16.183377547 +0000 UTC m=+2351.020431348" watchObservedRunningTime="2026-07-16 22:28:16.185314714 +0000 UTC m=+2351.022368556"
Jul 16 22:28:17.173338 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:17.173307    2555 generic.go:356] "Generic (PLEG): container finished" podID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerID="c5dcacf91ad79f09dfbd54f873767098677f88d2a06557859b8d849ea406e612" exitCode=6
Jul 16 22:28:17.173768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:17.173378    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerDied","Data":"c5dcacf91ad79f09dfbd54f873767098677f88d2a06557859b8d849ea406e612"}
Jul 16 22:28:17.173768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:17.173435    2555 scope.go:117] "RemoveContainer" containerID="ae2e6bbc900b1392999842174f4c6d9e79267f2adf8492be23338600ed3fe00c"
Jul 16 22:28:17.173768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:17.173702    2555 scope.go:117] "RemoveContainer" containerID="c5dcacf91ad79f09dfbd54f873767098677f88d2a06557859b8d849ea406e612"
Jul 16 22:28:17.173940 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:17.173921    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55_odh-ai-gateway-infra(359ed37e-d534-47dd-a1b7-66b553bc2efc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc"
Jul 16 22:28:17.199850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:17.199826    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:28:18.322117 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:18.322080    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:28:18.417832 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:18.417794    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca\") pod \"359ed37e-d534-47dd-a1b7-66b553bc2efc\" (UID: \"359ed37e-d534-47dd-a1b7-66b553bc2efc\") "
Jul 16 22:28:18.418175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:18.418152    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "359ed37e-d534-47dd-a1b7-66b553bc2efc" (UID: "359ed37e-d534-47dd-a1b7-66b553bc2efc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:18.518376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:18.518295    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/359ed37e-d534-47dd-a1b7-66b553bc2efc-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:28:19.187332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.187304    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"
Jul 16 22:28:19.187551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.187333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55" event={"ID":"359ed37e-d534-47dd-a1b7-66b553bc2efc","Type":"ContainerDied","Data":"3c0a5b052e03c863dd80c3d9fb7ee675365b5eef09329b37a427c86d1a98008e"}
Jul 16 22:28:19.187551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.187393    2555 scope.go:117] "RemoveContainer" containerID="c5dcacf91ad79f09dfbd54f873767098677f88d2a06557859b8d849ea406e612"
Jul 16 22:28:19.208003 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.207978    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:28:19.211406 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.211381    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wqg55"]
Jul 16 22:28:19.676887 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:19.676854    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" path="/var/lib/kubelet/pods/359ed37e-d534-47dd-a1b7-66b553bc2efc/volumes"
Jul 16 22:28:20.565223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565189    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:20.565565 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565551    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565565 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565567    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565576    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565582    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565588    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565593    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565608    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565613    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565628    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:28:20.565666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565633    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:28:20.565932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565680    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="08d6148e-5a1f-487d-a6af-0b3dc3b4704b" containerName="revoke-keys"
Jul 16 22:28:20.565932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565686    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9598449c-193b-478f-b2f3-b1a8bd5f29da" containerName="revoke-keys"
Jul 16 22:28:20.565932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565695    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.565932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565700    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1ddb245-8a5d-4fb8-8f26-3864c5e8ad37" containerName="revoke-keys"
Jul 16 22:28:20.565932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.565707    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:20.570242 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.570225    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:20.572844 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.572814    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:20.573528 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.573506    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:28:20.575672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.575652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:20.638939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.638903    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-h49pb\" (UID: \"0ed30af8-0312-445f-9e38-5430272802da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:20.739930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.739896    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-h49pb\" (UID: \"0ed30af8-0312-445f-9e38-5430272802da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:20.740537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.740519    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-h49pb\" (UID: \"0ed30af8-0312-445f-9e38-5430272802da\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:20.882481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:20.882370    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:21.015975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:21.015946    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:21.017524 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:28:21.017497    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ed30af8_0312_445f_9e38_5430272802da.slice/crio-2525c03cc7312a4ca6e209c0ad59dbadc75d60b43cabd296b80fa711873bd7ff WatchSource:0}: Error finding container 2525c03cc7312a4ca6e209c0ad59dbadc75d60b43cabd296b80fa711873bd7ff: Status 404 returned error can't find the container with id 2525c03cc7312a4ca6e209c0ad59dbadc75d60b43cabd296b80fa711873bd7ff
Jul 16 22:28:21.207305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:21.207232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerStarted","Data":"2525c03cc7312a4ca6e209c0ad59dbadc75d60b43cabd296b80fa711873bd7ff"}
Jul 16 22:28:22.213985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:22.213950    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed30af8-0312-445f-9e38-5430272802da" containerID="ae0b6c0fe8aad65b68dc04c4c2f3c58399df7044308225098c4096b3e985087d" exitCode=6
Jul 16 22:28:22.214487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:22.214005    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerDied","Data":"ae0b6c0fe8aad65b68dc04c4c2f3c58399df7044308225098c4096b3e985087d"}
Jul 16 22:28:22.214487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:22.214246    2555 scope.go:117] "RemoveContainer" containerID="ae0b6c0fe8aad65b68dc04c4c2f3c58399df7044308225098c4096b3e985087d"
Jul 16 22:28:23.220638 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:23.220547    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed30af8-0312-445f-9e38-5430272802da" containerID="3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103" exitCode=6
Jul 16 22:28:23.221257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:23.220643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerDied","Data":"3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103"}
Jul 16 22:28:23.221257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:23.220704    2555 scope.go:117] "RemoveContainer" containerID="ae0b6c0fe8aad65b68dc04c4c2f3c58399df7044308225098c4096b3e985087d"
Jul 16 22:28:23.221257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:23.220811    2555 scope.go:117] "RemoveContainer" containerID="3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103"
Jul 16 22:28:23.221257 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:23.221029    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-h49pb_odh-ai-gateway-infra(0ed30af8-0312-445f-9e38-5430272802da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" podUID="0ed30af8-0312-445f-9e38-5430272802da"
Jul 16 22:28:24.227425 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:24.227395    2555 scope.go:117] "RemoveContainer" containerID="3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103"
Jul 16 22:28:24.227843 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:24.227600    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-h49pb_odh-ai-gateway-infra(0ed30af8-0312-445f-9e38-5430272802da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" podUID="0ed30af8-0312-445f-9e38-5430272802da"
Jul 16 22:28:30.025483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.025429    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:30.025867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.025849    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="359ed37e-d534-47dd-a1b7-66b553bc2efc" containerName="revoke-keys"
Jul 16 22:28:30.029949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.029931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:30.036399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.036375    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:30.119791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.119756    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf\" (UID: \"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:30.220957 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.220914    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf\" (UID: \"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:30.221671 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.221646    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf\" (UID: \"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:30.340981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.340934    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:30.479164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:30.479139    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:30.481494 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:28:30.481442    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fdb8fbb_c819_43fe_b056_8fbd3a1b3f80.slice/crio-4b2b9da815f67d43c1bbc67a1b3f1889675578001b719c9ba6d574cdcb5cb8f6 WatchSource:0}: Error finding container 4b2b9da815f67d43c1bbc67a1b3f1889675578001b719c9ba6d574cdcb5cb8f6: Status 404 returned error can't find the container with id 4b2b9da815f67d43c1bbc67a1b3f1889675578001b719c9ba6d574cdcb5cb8f6
Jul 16 22:28:31.265316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:31.265277    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerID="d268f059d1d4612e49c9b284b60ce2dda4d8c2102acc879a3a387e68866a8fee" exitCode=6
Jul 16 22:28:31.265811 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:31.265357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" event={"ID":"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80","Type":"ContainerDied","Data":"d268f059d1d4612e49c9b284b60ce2dda4d8c2102acc879a3a387e68866a8fee"}
Jul 16 22:28:31.265811 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:31.265414    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" event={"ID":"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80","Type":"ContainerStarted","Data":"4b2b9da815f67d43c1bbc67a1b3f1889675578001b719c9ba6d574cdcb5cb8f6"}
Jul 16 22:28:31.265811 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:31.265600    2555 scope.go:117] "RemoveContainer" containerID="d268f059d1d4612e49c9b284b60ce2dda4d8c2102acc879a3a387e68866a8fee"
Jul 16 22:28:32.272027 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:32.271931    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerID="dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1" exitCode=6
Jul 16 22:28:32.272422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:32.272019    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" event={"ID":"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80","Type":"ContainerDied","Data":"dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1"}
Jul 16 22:28:32.272422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:32.272082    2555 scope.go:117] "RemoveContainer" containerID="d268f059d1d4612e49c9b284b60ce2dda4d8c2102acc879a3a387e68866a8fee"
Jul 16 22:28:32.272422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:32.272174    2555 scope.go:117] "RemoveContainer" containerID="dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1"
Jul 16 22:28:32.272422 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:32.272417    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf_odh-ai-gateway-infra(6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80"
Jul 16 22:28:33.280296 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:33.280273    2555 scope.go:117] "RemoveContainer" containerID="dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1"
Jul 16 22:28:33.280696 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:33.280544    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf_odh-ai-gateway-infra(6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80"
Jul 16 22:28:37.670516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:37.670457    2555 scope.go:117] "RemoveContainer" containerID="3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103"
Jul 16 22:28:38.306785 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.306752    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerStarted","Data":"67d6237bb2029d8a7edd7cf4fd46417d4a79bfa7a1ee87e9432ea36b7e25d546"}
Jul 16 22:28:38.333293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.333236    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" podStartSLOduration=18.333221431 podStartE2EDuration="18.333221431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:38.330604235 +0000 UTC m=+2373.167658045" watchObservedRunningTime="2026-07-16 22:28:38.333221431 +0000 UTC m=+2373.170275231"
Jul 16 22:28:38.702555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.702471    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:38.706001 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.705981    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:38.714558 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.714535    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:38.800889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.800842    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6\" (UID: \"30aca942-5082-450b-9d60-9a66c5742c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:38.902179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.902140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6\" (UID: \"30aca942-5082-450b-9d60-9a66c5742c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:38.902808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:38.902782    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6\" (UID: \"30aca942-5082-450b-9d60-9a66c5742c2b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:39.018548 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.018416    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:39.157141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.157076    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:39.159322 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:28:39.159291    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30aca942_5082_450b_9d60_9a66c5742c2b.slice/crio-73cf5913340e372ada0dce41556eb4a2f231a8a0e7353127da867d20d7ab2abd WatchSource:0}: Error finding container 73cf5913340e372ada0dce41556eb4a2f231a8a0e7353127da867d20d7ab2abd: Status 404 returned error can't find the container with id 73cf5913340e372ada0dce41556eb4a2f231a8a0e7353127da867d20d7ab2abd
Jul 16 22:28:39.313494 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.313442    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerStarted","Data":"73cf5913340e372ada0dce41556eb4a2f231a8a0e7353127da867d20d7ab2abd"}
Jul 16 22:28:39.315189 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.315163    2555 generic.go:356] "Generic (PLEG): container finished" podID="0ed30af8-0312-445f-9e38-5430272802da" containerID="67d6237bb2029d8a7edd7cf4fd46417d4a79bfa7a1ee87e9432ea36b7e25d546" exitCode=6
Jul 16 22:28:39.315304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.315201    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerDied","Data":"67d6237bb2029d8a7edd7cf4fd46417d4a79bfa7a1ee87e9432ea36b7e25d546"}
Jul 16 22:28:39.315304 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.315244    2555 scope.go:117] "RemoveContainer" containerID="3c413193283280bb1649ad30176b1f7e1b5a51bd6ecf49c3309766b62f714103"
Jul 16 22:28:39.315561 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.315546    2555 scope.go:117] "RemoveContainer" containerID="67d6237bb2029d8a7edd7cf4fd46417d4a79bfa7a1ee87e9432ea36b7e25d546"
Jul 16 22:28:39.315795 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:39.315775    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-h49pb_odh-ai-gateway-infra(0ed30af8-0312-445f-9e38-5430272802da)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" podUID="0ed30af8-0312-445f-9e38-5430272802da"
Jul 16 22:28:39.340994 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:39.340965    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:40.322379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.322338    2555 generic.go:356] "Generic (PLEG): container finished" podID="30aca942-5082-450b-9d60-9a66c5742c2b" containerID="4ea0879eef3dd589339af65187bdcdb25d6edaeab561fd6d84ec71ecd1e4ee67" exitCode=7
Jul 16 22:28:40.322846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.322417    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerDied","Data":"4ea0879eef3dd589339af65187bdcdb25d6edaeab561fd6d84ec71ecd1e4ee67"}
Jul 16 22:28:40.322846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.322632    2555 scope.go:117] "RemoveContainer" containerID="4ea0879eef3dd589339af65187bdcdb25d6edaeab561fd6d84ec71ecd1e4ee67"
Jul 16 22:28:40.459342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.459320    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:40.617799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.617773    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca\") pod \"0ed30af8-0312-445f-9e38-5430272802da\" (UID: \"0ed30af8-0312-445f-9e38-5430272802da\") "
Jul 16 22:28:40.618076 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.618051    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ed30af8-0312-445f-9e38-5430272802da" (UID: "0ed30af8-0312-445f-9e38-5430272802da"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:40.719027 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:40.718991    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ed30af8-0312-445f-9e38-5430272802da-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:28:41.331908 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.331874    2555 generic.go:356] "Generic (PLEG): container finished" podID="30aca942-5082-450b-9d60-9a66c5742c2b" containerID="477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886" exitCode=7
Jul 16 22:28:41.332398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.331966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerDied","Data":"477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886"}
Jul 16 22:28:41.332398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.332064    2555 scope.go:117] "RemoveContainer" containerID="4ea0879eef3dd589339af65187bdcdb25d6edaeab561fd6d84ec71ecd1e4ee67"
Jul 16 22:28:41.332398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.332171    2555 scope.go:117] "RemoveContainer" containerID="477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886"
Jul 16 22:28:41.332557 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:41.332417    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6_odh-ai-gateway-infra(30aca942-5082-450b-9d60-9a66c5742c2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" podUID="30aca942-5082-450b-9d60-9a66c5742c2b"
Jul 16 22:28:41.333375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.333357    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"
Jul 16 22:28:41.333489 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.333384    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb" event={"ID":"0ed30af8-0312-445f-9e38-5430272802da","Type":"ContainerDied","Data":"2525c03cc7312a4ca6e209c0ad59dbadc75d60b43cabd296b80fa711873bd7ff"}
Jul 16 22:28:41.343406 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.343390    2555 scope.go:117] "RemoveContainer" containerID="67d6237bb2029d8a7edd7cf4fd46417d4a79bfa7a1ee87e9432ea36b7e25d546"
Jul 16 22:28:41.360650 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.360625    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:41.363948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.363928    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-h49pb"]
Jul 16 22:28:41.678381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:41.678298    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ed30af8-0312-445f-9e38-5430272802da" path="/var/lib/kubelet/pods/0ed30af8-0312-445f-9e38-5430272802da/volumes"
Jul 16 22:28:42.340879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:42.340852    2555 scope.go:117] "RemoveContainer" containerID="477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886"
Jul 16 22:28:42.341240 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:42.341026    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6_odh-ai-gateway-infra(30aca942-5082-450b-9d60-9a66c5742c2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" podUID="30aca942-5082-450b-9d60-9a66c5742c2b"
Jul 16 22:28:45.673314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:45.673281    2555 scope.go:117] "RemoveContainer" containerID="dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1"
Jul 16 22:28:46.363132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:46.363101    2555 generic.go:356] "Generic (PLEG): container finished" podID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerID="50a5c4ecb89123dc56d638b47586f4ec3d072f5057dcd9b31ec011664c63a97f" exitCode=6
Jul 16 22:28:46.363289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:46.363149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" event={"ID":"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80","Type":"ContainerDied","Data":"50a5c4ecb89123dc56d638b47586f4ec3d072f5057dcd9b31ec011664c63a97f"}
Jul 16 22:28:46.363289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:46.363229    2555 scope.go:117] "RemoveContainer" containerID="dbb3d61772cfdac3e0339a5f2cd2905d1835ffa15005b4b06e2ab4c98a6278c1"
Jul 16 22:28:46.363622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:46.363604    2555 scope.go:117] "RemoveContainer" containerID="50a5c4ecb89123dc56d638b47586f4ec3d072f5057dcd9b31ec011664c63a97f"
Jul 16 22:28:46.363868 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:46.363847    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf_odh-ai-gateway-infra(6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80"
Jul 16 22:28:47.393061 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:47.393028    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:47.533784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:47.533756    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:47.576723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:47.576688    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca\") pod \"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80\" (UID: \"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80\") "
Jul 16 22:28:47.577043 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:47.577017    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" (UID: "6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:47.677648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:47.677555    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:28:48.378768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:48.378740    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"
Jul 16 22:28:48.378939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:48.378783    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf" event={"ID":"6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80","Type":"ContainerDied","Data":"4b2b9da815f67d43c1bbc67a1b3f1889675578001b719c9ba6d574cdcb5cb8f6"}
Jul 16 22:28:48.378939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:48.378847    2555 scope.go:117] "RemoveContainer" containerID="50a5c4ecb89123dc56d638b47586f4ec3d072f5057dcd9b31ec011664c63a97f"
Jul 16 22:28:48.399063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:48.399041    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:48.407849 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:48.407826    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-8h5nf"]
Jul 16 22:28:49.676333 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:49.676300    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" path="/var/lib/kubelet/pods/6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80/volumes"
Jul 16 22:28:51.492796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:51.492756    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:28:53.670926 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:53.670895    2555 scope.go:117] "RemoveContainer" containerID="477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886"
Jul 16 22:28:54.413815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:54.413780    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerStarted","Data":"56016f1cb24fd9b377353f7daccada1f5a076db7443b7bcdfa7002e1306da380"}
Jul 16 22:28:54.432595 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:54.432536    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" podStartSLOduration=16.432517206 podStartE2EDuration="16.432517206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:28:54.431950979 +0000 UTC m=+2389.269004779" watchObservedRunningTime="2026-07-16 22:28:54.432517206 +0000 UTC m=+2389.269571017"
Jul 16 22:28:55.421275 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:55.421240    2555 generic.go:356] "Generic (PLEG): container finished" podID="30aca942-5082-450b-9d60-9a66c5742c2b" containerID="56016f1cb24fd9b377353f7daccada1f5a076db7443b7bcdfa7002e1306da380" exitCode=6
Jul 16 22:28:55.421775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:55.421295    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerDied","Data":"56016f1cb24fd9b377353f7daccada1f5a076db7443b7bcdfa7002e1306da380"}
Jul 16 22:28:55.421775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:55.421360    2555 scope.go:117] "RemoveContainer" containerID="477c31abc1ad3497d0bc43c702c7692396ad8b62fea830b9faab15d7a66c8886"
Jul 16 22:28:55.421775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:55.421665    2555 scope.go:117] "RemoveContainer" containerID="56016f1cb24fd9b377353f7daccada1f5a076db7443b7bcdfa7002e1306da380"
Jul 16 22:28:55.421905 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:28:55.421863    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6_odh-ai-gateway-infra(30aca942-5082-450b-9d60-9a66c5742c2b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" podUID="30aca942-5082-450b-9d60-9a66c5742c2b"
Jul 16 22:28:55.451208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:55.451177    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:56.567166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:56.567140    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:56.656051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:56.656014    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca\") pod \"30aca942-5082-450b-9d60-9a66c5742c2b\" (UID: \"30aca942-5082-450b-9d60-9a66c5742c2b\") "
Jul 16 22:28:56.656389 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:56.656364    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "30aca942-5082-450b-9d60-9a66c5742c2b" (UID: "30aca942-5082-450b-9d60-9a66c5742c2b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:28:56.757559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:56.757457    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30aca942-5082-450b-9d60-9a66c5742c2b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:28:57.441666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.441635    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6" event={"ID":"30aca942-5082-450b-9d60-9a66c5742c2b","Type":"ContainerDied","Data":"73cf5913340e372ada0dce41556eb4a2f231a8a0e7353127da867d20d7ab2abd"}
Jul 16 22:28:57.441876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.441692    2555 scope.go:117] "RemoveContainer" containerID="56016f1cb24fd9b377353f7daccada1f5a076db7443b7bcdfa7002e1306da380"
Jul 16 22:28:57.441876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.441642    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"
Jul 16 22:28:57.467448 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.467414    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:57.471891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.471866    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-78wk6"]
Jul 16 22:28:57.676107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:57.676076    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" path="/var/lib/kubelet/pods/30aca942-5082-450b-9d60-9a66c5742c2b/volumes"
Jul 16 22:28:58.880655 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.880621    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:28:58.881133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881100    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881136    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881145    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881154    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881162    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881173    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881181    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881198    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881206    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881216    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881224    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881233    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881241    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881340    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881352    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881364    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881374    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881384    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881392    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881402    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ed30af8-0312-445f-9e38-5430272802da" containerName="revoke-keys"
Jul 16 22:28:58.881851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.881410    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:28:58.884693 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.884671    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:28:58.888162 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.888144    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:28:58.890250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.890229    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:28:58.897939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.897912    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:28:58.978036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:58.977993    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr\" (UID: \"3483ed66-8eef-4fe6-b509-4375595eee92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:28:59.078808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:59.078772    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr\" (UID: \"3483ed66-8eef-4fe6-b509-4375595eee92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:28:59.079443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:59.079419    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr\" (UID: \"3483ed66-8eef-4fe6-b509-4375595eee92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:28:59.195902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:59.195804    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:28:59.345142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:59.345107    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:28:59.346762 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:28:59.346736    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3483ed66_8eef_4fe6_b509_4375595eee92.slice/crio-9860fd3555913b66054c048051ef96bd9542f75d50f792c3e970d93d5b63255f WatchSource:0}: Error finding container 9860fd3555913b66054c048051ef96bd9542f75d50f792c3e970d93d5b63255f: Status 404 returned error can't find the container with id 9860fd3555913b66054c048051ef96bd9542f75d50f792c3e970d93d5b63255f
Jul 16 22:28:59.453643 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:28:59.453555    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" event={"ID":"3483ed66-8eef-4fe6-b509-4375595eee92","Type":"ContainerStarted","Data":"9860fd3555913b66054c048051ef96bd9542f75d50f792c3e970d93d5b63255f"}
Jul 16 22:29:00.460038 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:00.460001    2555 generic.go:356] "Generic (PLEG): container finished" podID="3483ed66-8eef-4fe6-b509-4375595eee92" containerID="b5664131971fc3927d9124eb0184d4aa79577e4d238f129a0a83c76e23944535" exitCode=6
Jul 16 22:29:00.460422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:00.460089    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" event={"ID":"3483ed66-8eef-4fe6-b509-4375595eee92","Type":"ContainerDied","Data":"b5664131971fc3927d9124eb0184d4aa79577e4d238f129a0a83c76e23944535"}
Jul 16 22:29:00.460422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:00.460282    2555 scope.go:117] "RemoveContainer" containerID="b5664131971fc3927d9124eb0184d4aa79577e4d238f129a0a83c76e23944535"
Jul 16 22:29:01.467186 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:01.467143    2555 generic.go:356] "Generic (PLEG): container finished" podID="3483ed66-8eef-4fe6-b509-4375595eee92" containerID="c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8" exitCode=6
Jul 16 22:29:01.467617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:01.467190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" event={"ID":"3483ed66-8eef-4fe6-b509-4375595eee92","Type":"ContainerDied","Data":"c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8"}
Jul 16 22:29:01.467617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:01.467279    2555 scope.go:117] "RemoveContainer" containerID="b5664131971fc3927d9124eb0184d4aa79577e4d238f129a0a83c76e23944535"
Jul 16 22:29:01.467617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:01.467427    2555 scope.go:117] "RemoveContainer" containerID="c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8"
Jul 16 22:29:01.467742 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:01.467705    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr_odh-ai-gateway-infra(3483ed66-8eef-4fe6-b509-4375595eee92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" podUID="3483ed66-8eef-4fe6-b509-4375595eee92"
Jul 16 22:29:02.474347 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:02.474314    2555 scope.go:117] "RemoveContainer" containerID="c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8"
Jul 16 22:29:02.474745 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:02.474539    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr_odh-ai-gateway-infra(3483ed66-8eef-4fe6-b509-4375595eee92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" podUID="3483ed66-8eef-4fe6-b509-4375595eee92"
Jul 16 22:29:16.671058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:16.671027    2555 scope.go:117] "RemoveContainer" containerID="c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8"
Jul 16 22:29:17.550137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:17.550106    2555 generic.go:356] "Generic (PLEG): container finished" podID="3483ed66-8eef-4fe6-b509-4375595eee92" containerID="8d91884db052fd2e79c6f4bc058a80b3162f2c938cba5d6a5321b9bfe7b7c1fc" exitCode=6
Jul 16 22:29:17.550307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:17.550176    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" event={"ID":"3483ed66-8eef-4fe6-b509-4375595eee92","Type":"ContainerDied","Data":"8d91884db052fd2e79c6f4bc058a80b3162f2c938cba5d6a5321b9bfe7b7c1fc"}
Jul 16 22:29:17.550307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:17.550238    2555 scope.go:117] "RemoveContainer" containerID="c56b57fb714d153b06f34e1aa6c1b8001b4dcc21a4758e01cd155a2afffcb2b8"
Jul 16 22:29:17.550550 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:17.550529    2555 scope.go:117] "RemoveContainer" containerID="8d91884db052fd2e79c6f4bc058a80b3162f2c938cba5d6a5321b9bfe7b7c1fc"
Jul 16 22:29:17.550787 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:17.550769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr_odh-ai-gateway-infra(3483ed66-8eef-4fe6-b509-4375595eee92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" podUID="3483ed66-8eef-4fe6-b509-4375595eee92"
Jul 16 22:29:18.582248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:18.582219    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:29:18.721776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:18.721747    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:29:18.843696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:18.843612    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca\") pod \"3483ed66-8eef-4fe6-b509-4375595eee92\" (UID: \"3483ed66-8eef-4fe6-b509-4375595eee92\") "
Jul 16 22:29:18.844024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:18.843993    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3483ed66-8eef-4fe6-b509-4375595eee92" (UID: "3483ed66-8eef-4fe6-b509-4375595eee92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:18.944786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:18.944747    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3483ed66-8eef-4fe6-b509-4375595eee92-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:29:19.563142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.563114    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"
Jul 16 22:29:19.563142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.563134    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr" event={"ID":"3483ed66-8eef-4fe6-b509-4375595eee92","Type":"ContainerDied","Data":"9860fd3555913b66054c048051ef96bd9542f75d50f792c3e970d93d5b63255f"}
Jul 16 22:29:19.563409 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.563217    2555 scope.go:117] "RemoveContainer" containerID="8d91884db052fd2e79c6f4bc058a80b3162f2c938cba5d6a5321b9bfe7b7c1fc"
Jul 16 22:29:19.588729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.588703    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:29:19.596979 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.596958    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5wnmr"]
Jul 16 22:29:19.675599 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:19.675572    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" path="/var/lib/kubelet/pods/3483ed66-8eef-4fe6-b509-4375595eee92/volumes"
Jul 16 22:29:20.063772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.063741    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:20.064089 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064076    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064092    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064110    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064116    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="30aca942-5082-450b-9d60-9a66c5742c2b" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064137    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064142    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064150    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064155    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064161    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064167    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064223    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064232    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6fdb8fbb-c819-43fe-b056-8fbd3a1b3f80" containerName="revoke-keys"
Jul 16 22:29:20.064546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064239    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.064546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.064245    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3483ed66-8eef-4fe6-b509-4375595eee92" containerName="revoke-keys"
Jul 16 22:29:20.068665 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.068642    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:20.071268 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.071242    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:29:20.071369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.071242    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:29:20.076965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.076945    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:20.156020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.155981    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r\" (UID: \"0f65d65f-a284-4794-b9d2-497f2f8ecc53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:20.257223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.257187    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r\" (UID: \"0f65d65f-a284-4794-b9d2-497f2f8ecc53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:20.257796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.257778    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r\" (UID: \"0f65d65f-a284-4794-b9d2-497f2f8ecc53\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:20.380809 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.380713    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:20.545891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.545856    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:20.548202 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:29:20.548173    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f65d65f_a284_4794_b9d2_497f2f8ecc53.slice/crio-03a315a9a4adc49f44171fa512b5a72dc7cb28b99929456195cf1ee9fcea0a41 WatchSource:0}: Error finding container 03a315a9a4adc49f44171fa512b5a72dc7cb28b99929456195cf1ee9fcea0a41: Status 404 returned error can't find the container with id 03a315a9a4adc49f44171fa512b5a72dc7cb28b99929456195cf1ee9fcea0a41
Jul 16 22:29:20.570124 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.570099    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" event={"ID":"0f65d65f-a284-4794-b9d2-497f2f8ecc53","Type":"ContainerStarted","Data":"03a315a9a4adc49f44171fa512b5a72dc7cb28b99929456195cf1ee9fcea0a41"}
Jul 16 22:29:20.637141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.637116    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:20.641787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.641769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:20.651975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.651948    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:20.764930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.764899    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-w47gd\" (UID: \"640085a7-3bc3-4a7e-80b9-9f994d1c63fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:20.865753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.865718    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-w47gd\" (UID: \"640085a7-3bc3-4a7e-80b9-9f994d1c63fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:20.866311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.866291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-w47gd\" (UID: \"640085a7-3bc3-4a7e-80b9-9f994d1c63fd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:20.951939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:20.951857    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:21.296936 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:21.296907    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:21.298286 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:29:21.298262    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod640085a7_3bc3_4a7e_80b9_9f994d1c63fd.slice/crio-0cda5fdf6a073a089f34400d950e91dea52248f3345b22d3f36ba09cf729e853 WatchSource:0}: Error finding container 0cda5fdf6a073a089f34400d950e91dea52248f3345b22d3f36ba09cf729e853: Status 404 returned error can't find the container with id 0cda5fdf6a073a089f34400d950e91dea52248f3345b22d3f36ba09cf729e853
Jul 16 22:29:21.577933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:21.577848    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" event={"ID":"640085a7-3bc3-4a7e-80b9-9f994d1c63fd","Type":"ContainerStarted","Data":"0cda5fdf6a073a089f34400d950e91dea52248f3345b22d3f36ba09cf729e853"}
Jul 16 22:29:21.579602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:21.579572    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerID="42ee0a57e16e82d19d7425af3c8cc0be9fb2e44b44489acae8609c9c04c95ac9" exitCode=6
Jul 16 22:29:21.579738 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:21.579618    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" event={"ID":"0f65d65f-a284-4794-b9d2-497f2f8ecc53","Type":"ContainerDied","Data":"42ee0a57e16e82d19d7425af3c8cc0be9fb2e44b44489acae8609c9c04c95ac9"}
Jul 16 22:29:21.579859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:21.579846    2555 scope.go:117] "RemoveContainer" containerID="42ee0a57e16e82d19d7425af3c8cc0be9fb2e44b44489acae8609c9c04c95ac9"
Jul 16 22:29:22.586202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.586114    2555 generic.go:356] "Generic (PLEG): container finished" podID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerID="44bf1f8bbb564ce8eb8306cfa092d9d90e097dd2487018ded7c3fe33db0fc587" exitCode=6
Jul 16 22:29:22.586692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.586198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" event={"ID":"640085a7-3bc3-4a7e-80b9-9f994d1c63fd","Type":"ContainerDied","Data":"44bf1f8bbb564ce8eb8306cfa092d9d90e097dd2487018ded7c3fe33db0fc587"}
Jul 16 22:29:22.586692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.586427    2555 scope.go:117] "RemoveContainer" containerID="44bf1f8bbb564ce8eb8306cfa092d9d90e097dd2487018ded7c3fe33db0fc587"
Jul 16 22:29:22.588125 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.588108    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerID="0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc" exitCode=6
Jul 16 22:29:22.588216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.588141    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" event={"ID":"0f65d65f-a284-4794-b9d2-497f2f8ecc53","Type":"ContainerDied","Data":"0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc"}
Jul 16 22:29:22.588216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.588205    2555 scope.go:117] "RemoveContainer" containerID="42ee0a57e16e82d19d7425af3c8cc0be9fb2e44b44489acae8609c9c04c95ac9"
Jul 16 22:29:22.588405 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:22.588386    2555 scope.go:117] "RemoveContainer" containerID="0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc"
Jul 16 22:29:22.588636 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:22.588607    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r_odh-ai-gateway-infra(0f65d65f-a284-4794-b9d2-497f2f8ecc53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53"
Jul 16 22:29:23.596136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:23.596092    2555 generic.go:356] "Generic (PLEG): container finished" podID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerID="cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299" exitCode=6
Jul 16 22:29:23.596642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:23.596175    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" event={"ID":"640085a7-3bc3-4a7e-80b9-9f994d1c63fd","Type":"ContainerDied","Data":"cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299"}
Jul 16 22:29:23.596642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:23.596281    2555 scope.go:117] "RemoveContainer" containerID="44bf1f8bbb564ce8eb8306cfa092d9d90e097dd2487018ded7c3fe33db0fc587"
Jul 16 22:29:23.596642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:23.596498    2555 scope.go:117] "RemoveContainer" containerID="cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299"
Jul 16 22:29:23.596817 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:23.596711    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-w47gd_odh-ai-gateway-infra(640085a7-3bc3-4a7e-80b9-9f994d1c63fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd"
Jul 16 22:29:23.598051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:23.598034    2555 scope.go:117] "RemoveContainer" containerID="0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc"
Jul 16 22:29:23.598193 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:23.598177    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r_odh-ai-gateway-infra(0f65d65f-a284-4794-b9d2-497f2f8ecc53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53"
Jul 16 22:29:24.606365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:24.606337    2555 scope.go:117] "RemoveContainer" containerID="cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299"
Jul 16 22:29:24.606761 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:24.606547    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-w47gd_odh-ai-gateway-infra(640085a7-3bc3-4a7e-80b9-9f994d1c63fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd"
Jul 16 22:29:37.670377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:37.670348    2555 scope.go:117] "RemoveContainer" containerID="cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299"
Jul 16 22:29:38.670496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:38.670383    2555 scope.go:117] "RemoveContainer" containerID="0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc"
Jul 16 22:29:38.685653 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:38.685623    2555 generic.go:356] "Generic (PLEG): container finished" podID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerID="ed7badcf4bdd0155ab51638e2f40afadf79e820c2b377d3d300e6a20d882cee0" exitCode=6
Jul 16 22:29:38.685784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:38.685689    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" event={"ID":"640085a7-3bc3-4a7e-80b9-9f994d1c63fd","Type":"ContainerDied","Data":"ed7badcf4bdd0155ab51638e2f40afadf79e820c2b377d3d300e6a20d882cee0"}
Jul 16 22:29:38.685784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:38.685746    2555 scope.go:117] "RemoveContainer" containerID="cf0412c0577c45daeac1097b50b7e20f4e8622bab71f68a746cd75b1ad0c5299"
Jul 16 22:29:38.686047 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:38.686027    2555 scope.go:117] "RemoveContainer" containerID="ed7badcf4bdd0155ab51638e2f40afadf79e820c2b377d3d300e6a20d882cee0"
Jul 16 22:29:38.686234 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:38.686213    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-w47gd_odh-ai-gateway-infra(640085a7-3bc3-4a7e-80b9-9f994d1c63fd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd"
Jul 16 22:29:39.691891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.691800    2555 generic.go:356] "Generic (PLEG): container finished" podID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerID="55b5be634a8bf64b93d41ef1e0c57e0e418dd656dac8ee6fc699de5cc3c825e2" exitCode=6
Jul 16 22:29:39.691891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.691874    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" event={"ID":"0f65d65f-a284-4794-b9d2-497f2f8ecc53","Type":"ContainerDied","Data":"55b5be634a8bf64b93d41ef1e0c57e0e418dd656dac8ee6fc699de5cc3c825e2"}
Jul 16 22:29:39.692378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.691934    2555 scope.go:117] "RemoveContainer" containerID="0f7192bf5164657942bdcbe7536924f200db030d5b2f5b44c270d41e30a404fc"
Jul 16 22:29:39.692378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.692242    2555 scope.go:117] "RemoveContainer" containerID="55b5be634a8bf64b93d41ef1e0c57e0e418dd656dac8ee6fc699de5cc3c825e2"
Jul 16 22:29:39.692458 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:29:39.692434    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r_odh-ai-gateway-infra(0f65d65f-a284-4794-b9d2-497f2f8ecc53)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53"
Jul 16 22:29:39.724880 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.724853    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:39.861074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.861054    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:39.927757 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.927723    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca\") pod \"640085a7-3bc3-4a7e-80b9-9f994d1c63fd\" (UID: \"640085a7-3bc3-4a7e-80b9-9f994d1c63fd\") "
Jul 16 22:29:39.928104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:39.928078    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "640085a7-3bc3-4a7e-80b9-9f994d1c63fd" (UID: "640085a7-3bc3-4a7e-80b9-9f994d1c63fd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:40.028637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.028602    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/640085a7-3bc3-4a7e-80b9-9f994d1c63fd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:29:40.700600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.700566    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"
Jul 16 22:29:40.701070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.700602    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd" event={"ID":"640085a7-3bc3-4a7e-80b9-9f994d1c63fd","Type":"ContainerDied","Data":"0cda5fdf6a073a089f34400d950e91dea52248f3345b22d3f36ba09cf729e853"}
Jul 16 22:29:40.701070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.700658    2555 scope.go:117] "RemoveContainer" containerID="ed7badcf4bdd0155ab51638e2f40afadf79e820c2b377d3d300e6a20d882cee0"
Jul 16 22:29:40.727634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.727606    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:40.733414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.733390    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:40.748370 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.748346    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-w47gd"]
Jul 16 22:29:40.864164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.864134    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:40.936554 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.936516    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca\") pod \"0f65d65f-a284-4794-b9d2-497f2f8ecc53\" (UID: \"0f65d65f-a284-4794-b9d2-497f2f8ecc53\") "
Jul 16 22:29:40.936899 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:40.936877    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0f65d65f-a284-4794-b9d2-497f2f8ecc53" (UID: "0f65d65f-a284-4794-b9d2-497f2f8ecc53"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:29:41.037146 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.037118    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f65d65f-a284-4794-b9d2-497f2f8ecc53-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:29:41.676777 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.676733    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" path="/var/lib/kubelet/pods/640085a7-3bc3-4a7e-80b9-9f994d1c63fd/volumes"
Jul 16 22:29:41.709923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.709895    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"
Jul 16 22:29:41.710381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.709920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r" event={"ID":"0f65d65f-a284-4794-b9d2-497f2f8ecc53","Type":"ContainerDied","Data":"03a315a9a4adc49f44171fa512b5a72dc7cb28b99929456195cf1ee9fcea0a41"}
Jul 16 22:29:41.710381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.709989    2555 scope.go:117] "RemoveContainer" containerID="55b5be634a8bf64b93d41ef1e0c57e0e418dd656dac8ee6fc699de5cc3c825e2"
Jul 16 22:29:41.732172 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.732131    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:41.738453 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:41.738426    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-f4x4r"]
Jul 16 22:29:43.675949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:43.675913    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" path="/var/lib/kubelet/pods/0f65d65f-a284-4794-b9d2-497f2f8ecc53/volumes"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936045    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936897    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936919    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936946    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936955    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936971    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.936981    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937002    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937011    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937044    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937053    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937205    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937218    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937227    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="640085a7-3bc3-4a7e-80b9-9f994d1c63fd" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937244    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937252    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.938905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.937269    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:29:58.942957 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.942933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:29:58.946164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.946025    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:29:58.946164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.946140    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:29:58.947979 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:58.947954    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:29:59.000887 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.000855    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b\" (UID: \"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:29:59.101533 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.101493    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b\" (UID: \"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:29:59.102122 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.102102    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b\" (UID: \"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:29:59.254421 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.254333    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:29:59.391985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.391807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:29:59.394268 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:29:59.394237    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode899fdc6_2d2c_4f06_a1b0_abd932bf01e0.slice/crio-548262dcfa5e5018e15bc80d417dc6796cacb78923640fd2e335ba7835ead51c WatchSource:0}: Error finding container 548262dcfa5e5018e15bc80d417dc6796cacb78923640fd2e335ba7835ead51c: Status 404 returned error can't find the container with id 548262dcfa5e5018e15bc80d417dc6796cacb78923640fd2e335ba7835ead51c
Jul 16 22:29:59.810010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.809973    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerStarted","Data":"c0813a239d5ad4922783a70e6bf6970cbd388ea12756b2058bc94fb87adecb6a"}
Jul 16 22:29:59.810181 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.810022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerStarted","Data":"548262dcfa5e5018e15bc80d417dc6796cacb78923640fd2e335ba7835ead51c"}
Jul 16 22:29:59.829703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:29:59.829657    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" podStartSLOduration=1.8296444429999998 podStartE2EDuration="1.829644443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:29:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:29:59.826725535 +0000 UTC m=+2454.663779347" watchObservedRunningTime="2026-07-16 22:29:59.829644443 +0000 UTC m=+2454.666698243"
Jul 16 22:30:00.817989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:00.817958    2555 generic.go:356] "Generic (PLEG): container finished" podID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerID="c0813a239d5ad4922783a70e6bf6970cbd388ea12756b2058bc94fb87adecb6a" exitCode=6
Jul 16 22:30:00.818395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:00.818001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerDied","Data":"c0813a239d5ad4922783a70e6bf6970cbd388ea12756b2058bc94fb87adecb6a"}
Jul 16 22:30:00.818395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:00.818187    2555 scope.go:117] "RemoveContainer" containerID="c0813a239d5ad4922783a70e6bf6970cbd388ea12756b2058bc94fb87adecb6a"
Jul 16 22:30:01.825446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:01.825414    2555 generic.go:356] "Generic (PLEG): container finished" podID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerID="aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef" exitCode=6
Jul 16 22:30:01.825963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:01.825494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerDied","Data":"aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef"}
Jul 16 22:30:01.825963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:01.825543    2555 scope.go:117] "RemoveContainer" containerID="c0813a239d5ad4922783a70e6bf6970cbd388ea12756b2058bc94fb87adecb6a"
Jul 16 22:30:01.825963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:01.825774    2555 scope.go:117] "RemoveContainer" containerID="aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef"
Jul 16 22:30:01.826129 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:01.825996    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b_odh-ai-gateway-infra(e899fdc6-2d2c-4f06-a1b0-abd932bf01e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0"
Jul 16 22:30:02.832416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:02.832388    2555 scope.go:117] "RemoveContainer" containerID="aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef"
Jul 16 22:30:02.832817 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:02.832589    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b_odh-ai-gateway-infra(e899fdc6-2d2c-4f06-a1b0-abd932bf01e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0"
Jul 16 22:30:16.671143 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:16.671107    2555 scope.go:117] "RemoveContainer" containerID="aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef"
Jul 16 22:30:17.912248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:17.912213    2555 generic.go:356] "Generic (PLEG): container finished" podID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerID="a6bddf37cca5ba061e1e3dd3f50952f2f7ad4034c0d8b2e93abbff6c6366e3f5" exitCode=6
Jul 16 22:30:17.912248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:17.912254    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerDied","Data":"a6bddf37cca5ba061e1e3dd3f50952f2f7ad4034c0d8b2e93abbff6c6366e3f5"}
Jul 16 22:30:17.912786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:17.912313    2555 scope.go:117] "RemoveContainer" containerID="aa4073bd498e791a8216aad3bd1996bf64601ddd7756e9f981a5eabd1df285ef"
Jul 16 22:30:17.912786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:17.912628    2555 scope.go:117] "RemoveContainer" containerID="a6bddf37cca5ba061e1e3dd3f50952f2f7ad4034c0d8b2e93abbff6c6366e3f5"
Jul 16 22:30:17.912900 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:17.912847    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b_odh-ai-gateway-infra(e899fdc6-2d2c-4f06-a1b0-abd932bf01e0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0"
Jul 16 22:30:18.971215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:18.971182    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:30:19.106866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.106835    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:30:19.285597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.285567    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca\") pod \"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0\" (UID: \"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0\") "
Jul 16 22:30:19.285924 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.285902    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" (UID: "e899fdc6-2d2c-4f06-a1b0-abd932bf01e0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:19.386983 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.386946    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:30:19.925790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.925763    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"
Jul 16 22:30:19.925962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.925803    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b" event={"ID":"e899fdc6-2d2c-4f06-a1b0-abd932bf01e0","Type":"ContainerDied","Data":"548262dcfa5e5018e15bc80d417dc6796cacb78923640fd2e335ba7835ead51c"}
Jul 16 22:30:19.925962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.925861    2555 scope.go:117] "RemoveContainer" containerID="a6bddf37cca5ba061e1e3dd3f50952f2f7ad4034c0d8b2e93abbff6c6366e3f5"
Jul 16 22:30:19.945186 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.945158    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:30:19.949995 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:19.949966    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-cjh5b"]
Jul 16 22:30:20.131268 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131230    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131587    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131598    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f65d65f-a284-4794-b9d2-497f2f8ecc53" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131609    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131615    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131622    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131628    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131694    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.131713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.131702    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:20.136203 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.136181    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:20.138961 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.138939    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:30:20.139092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.139069    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:30:20.144720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.144696    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:20.194714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.194627    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7\" (UID: \"289e4e4a-f1f5-4db2-8739-17080a098edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:20.295146 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.295099    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7\" (UID: \"289e4e4a-f1f5-4db2-8739-17080a098edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:20.295934 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.295908    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7\" (UID: \"289e4e4a-f1f5-4db2-8739-17080a098edf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:20.449202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.449115    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:20.796359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.796328    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:20.797646 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:30:20.797616    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289e4e4a_f1f5_4db2_8739_17080a098edf.slice/crio-22d7aa55f52004a8ee71145f89aad906ce64b705ecec1a2f598dc3402a5bf20b WatchSource:0}: Error finding container 22d7aa55f52004a8ee71145f89aad906ce64b705ecec1a2f598dc3402a5bf20b: Status 404 returned error can't find the container with id 22d7aa55f52004a8ee71145f89aad906ce64b705ecec1a2f598dc3402a5bf20b
Jul 16 22:30:20.934221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:20.934178    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerStarted","Data":"22d7aa55f52004a8ee71145f89aad906ce64b705ecec1a2f598dc3402a5bf20b"}
Jul 16 22:30:21.676770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:21.676741    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" path="/var/lib/kubelet/pods/e899fdc6-2d2c-4f06-a1b0-abd932bf01e0/volumes"
Jul 16 22:30:21.940802 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:21.940768    2555 generic.go:356] "Generic (PLEG): container finished" podID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerID="31cc90176469d0175691059dbf25af3977a8c43b8a1b60f637acb7e3dbc7701a" exitCode=6
Jul 16 22:30:21.940962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:21.940839    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerDied","Data":"31cc90176469d0175691059dbf25af3977a8c43b8a1b60f637acb7e3dbc7701a"}
Jul 16 22:30:21.941024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:21.941008    2555 scope.go:117] "RemoveContainer" containerID="31cc90176469d0175691059dbf25af3977a8c43b8a1b60f637acb7e3dbc7701a"
Jul 16 22:30:22.543695 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:22.543662    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod289e4e4a_f1f5_4db2_8739_17080a098edf.slice/crio-3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:30:22.947787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:22.947753    2555 generic.go:356] "Generic (PLEG): container finished" podID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerID="3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741" exitCode=6
Jul 16 22:30:22.948277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:22.947840    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerDied","Data":"3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741"}
Jul 16 22:30:22.948277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:22.947948    2555 scope.go:117] "RemoveContainer" containerID="31cc90176469d0175691059dbf25af3977a8c43b8a1b60f637acb7e3dbc7701a"
Jul 16 22:30:22.948277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:22.948000    2555 scope.go:117] "RemoveContainer" containerID="3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741"
Jul 16 22:30:22.948277 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:22.948229    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7_odh-ai-gateway-infra(289e4e4a-f1f5-4db2-8739-17080a098edf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf"
Jul 16 22:30:23.954894 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:23.954865    2555 scope.go:117] "RemoveContainer" containerID="3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741"
Jul 16 22:30:23.955303 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:23.955044    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7_odh-ai-gateway-infra(289e4e4a-f1f5-4db2-8739-17080a098edf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf"
Jul 16 22:30:27.632267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:27.632233    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:30:28.853369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.853328    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:28.853758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.853732    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:28.853758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.853744    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:28.853830 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.853797    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e899fdc6-2d2c-4f06-a1b0-abd932bf01e0" containerName="revoke-keys"
Jul 16 22:30:28.858067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.858049    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:28.858405 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.858380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq\" (UID: \"5a2d2a6c-a3f3-40fe-a993-c72529144b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:28.876068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.876044    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:28.959534 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.959497    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq\" (UID: \"5a2d2a6c-a3f3-40fe-a993-c72529144b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:28.960185 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:28.960158    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq\" (UID: \"5a2d2a6c-a3f3-40fe-a993-c72529144b1a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:29.168183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.168086    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:29.323171 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.323146    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:29.325141 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:30:29.325115    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a2d2a6c_a3f3_40fe_a993_c72529144b1a.slice/crio-919b5d15f782190f29f2294c51c47a9946d1c9fa7d76e1afa50dfa06adcdfe87 WatchSource:0}: Error finding container 919b5d15f782190f29f2294c51c47a9946d1c9fa7d76e1afa50dfa06adcdfe87: Status 404 returned error can't find the container with id 919b5d15f782190f29f2294c51c47a9946d1c9fa7d76e1afa50dfa06adcdfe87
Jul 16 22:30:29.986221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.986185    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerID="e1da577fe7cd472ef6f36f0dca9cadde1e6cd1f6831c48396b423a6bd17903cb" exitCode=6
Jul 16 22:30:29.986631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.986251    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerDied","Data":"e1da577fe7cd472ef6f36f0dca9cadde1e6cd1f6831c48396b423a6bd17903cb"}
Jul 16 22:30:29.986631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.986298    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerStarted","Data":"919b5d15f782190f29f2294c51c47a9946d1c9fa7d76e1afa50dfa06adcdfe87"}
Jul 16 22:30:29.986631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:29.986406    2555 scope.go:117] "RemoveContainer" containerID="e1da577fe7cd472ef6f36f0dca9cadde1e6cd1f6831c48396b423a6bd17903cb"
Jul 16 22:30:30.994568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:30.994449    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerID="fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b" exitCode=6
Jul 16 22:30:30.994568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:30.994542    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerDied","Data":"fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b"}
Jul 16 22:30:30.995057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:30.994610    2555 scope.go:117] "RemoveContainer" containerID="e1da577fe7cd472ef6f36f0dca9cadde1e6cd1f6831c48396b423a6bd17903cb"
Jul 16 22:30:30.995057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:30.994777    2555 scope.go:117] "RemoveContainer" containerID="fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b"
Jul 16 22:30:30.995057 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:30.995040    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq_odh-ai-gateway-infra(5a2d2a6c-a3f3-40fe-a993-c72529144b1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a"
Jul 16 22:30:32.007689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:32.007659    2555 scope.go:117] "RemoveContainer" containerID="fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b"
Jul 16 22:30:32.008080 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:32.007860    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq_odh-ai-gateway-infra(5a2d2a6c-a3f3-40fe-a993-c72529144b1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a"
Jul 16 22:30:34.671063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:34.671028    2555 scope.go:117] "RemoveContainer" containerID="3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741"
Jul 16 22:30:35.026372 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:35.026294    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerStarted","Data":"dc9cff3aae800ae83e9a9748f35ec69b600acaa29ca36f5ee7ad7af207180f30"}
Jul 16 22:30:35.067365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:35.067316    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" podStartSLOduration=15.067300634 podStartE2EDuration="15.067300634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:30:35.066332627 +0000 UTC m=+2489.903386429" watchObservedRunningTime="2026-07-16 22:30:35.067300634 +0000 UTC m=+2489.904354458"
Jul 16 22:30:36.033521 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:36.033481    2555 generic.go:356] "Generic (PLEG): container finished" podID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerID="dc9cff3aae800ae83e9a9748f35ec69b600acaa29ca36f5ee7ad7af207180f30" exitCode=6
Jul 16 22:30:36.033521 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:36.033504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerDied","Data":"dc9cff3aae800ae83e9a9748f35ec69b600acaa29ca36f5ee7ad7af207180f30"}
Jul 16 22:30:36.034014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:36.033569    2555 scope.go:117] "RemoveContainer" containerID="3dd932ae5178e307e82565f7a81f7dc5c389a17f0b0f0626e426fe69044b3741"
Jul 16 22:30:36.034014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:36.033841    2555 scope.go:117] "RemoveContainer" containerID="dc9cff3aae800ae83e9a9748f35ec69b600acaa29ca36f5ee7ad7af207180f30"
Jul 16 22:30:36.034122 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:36.034048    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7_odh-ai-gateway-infra(289e4e4a-f1f5-4db2-8739-17080a098edf)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf"
Jul 16 22:30:36.086599 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:36.086565    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:37.181247 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:37.181223    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:37.228770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:37.228733    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca\") pod \"289e4e4a-f1f5-4db2-8739-17080a098edf\" (UID: \"289e4e4a-f1f5-4db2-8739-17080a098edf\") "
Jul 16 22:30:37.229104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:37.229083    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "289e4e4a-f1f5-4db2-8739-17080a098edf" (UID: "289e4e4a-f1f5-4db2-8739-17080a098edf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:37.329582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:37.329459    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/289e4e4a-f1f5-4db2-8739-17080a098edf-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:30:38.047589 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.047555    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"
Jul 16 22:30:38.047797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.047594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7" event={"ID":"289e4e4a-f1f5-4db2-8739-17080a098edf","Type":"ContainerDied","Data":"22d7aa55f52004a8ee71145f89aad906ce64b705ecec1a2f598dc3402a5bf20b"}
Jul 16 22:30:38.047797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.047697    2555 scope.go:117] "RemoveContainer" containerID="dc9cff3aae800ae83e9a9748f35ec69b600acaa29ca36f5ee7ad7af207180f30"
Jul 16 22:30:38.064640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.064615    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:38.068075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.068053    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-97cz7"]
Jul 16 22:30:38.093584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:38.093552    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:30:39.677017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:39.676978    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" path="/var/lib/kubelet/pods/289e4e4a-f1f5-4db2-8739-17080a098edf/volumes"
Jul 16 22:30:46.671118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:46.671091    2555 scope.go:117] "RemoveContainer" containerID="fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b"
Jul 16 22:30:47.101546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:47.101513    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerStarted","Data":"48e03ef0bee7a8a8c747e94890137464572b515eabf812317e882a652fbfb77d"}
Jul 16 22:30:48.109042 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:48.109008    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerID="48e03ef0bee7a8a8c747e94890137464572b515eabf812317e882a652fbfb77d" exitCode=6
Jul 16 22:30:48.109452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:48.109084    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerDied","Data":"48e03ef0bee7a8a8c747e94890137464572b515eabf812317e882a652fbfb77d"}
Jul 16 22:30:48.109452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:48.109148    2555 scope.go:117] "RemoveContainer" containerID="fa37d4feaa5b4d64b8c4de891b83949d378b2c71e3276f1951ba4455d2a97e3b"
Jul 16 22:30:48.109452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:48.109380    2555 scope.go:117] "RemoveContainer" containerID="48e03ef0bee7a8a8c747e94890137464572b515eabf812317e882a652fbfb77d"
Jul 16 22:30:48.109683 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:30:48.109659    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq_odh-ai-gateway-infra(5a2d2a6c-a3f3-40fe-a993-c72529144b1a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a"
Jul 16 22:30:48.175478 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:48.175438    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:49.251514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:49.251490    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:49.334175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:49.334140    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca\") pod \"5a2d2a6c-a3f3-40fe-a993-c72529144b1a\" (UID: \"5a2d2a6c-a3f3-40fe-a993-c72529144b1a\") "
Jul 16 22:30:49.334572 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:49.334551    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a2d2a6c-a3f3-40fe-a993-c72529144b1a" (UID: "5a2d2a6c-a3f3-40fe-a993-c72529144b1a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:30:49.435628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:49.435546    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a2d2a6c-a3f3-40fe-a993-c72529144b1a-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:30:50.122506 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:50.122445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq" event={"ID":"5a2d2a6c-a3f3-40fe-a993-c72529144b1a","Type":"ContainerDied","Data":"919b5d15f782190f29f2294c51c47a9946d1c9fa7d76e1afa50dfa06adcdfe87"}
Jul 16 22:30:50.122666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:50.122535    2555 scope.go:117] "RemoveContainer" containerID="48e03ef0bee7a8a8c747e94890137464572b515eabf812317e882a652fbfb77d"
Jul 16 22:30:50.122666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:50.122451    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"
Jul 16 22:30:50.141215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:50.141186    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:50.149497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:50.149454    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fkpcq"]
Jul 16 22:30:51.676717 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:51.676673    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" path="/var/lib/kubelet/pods/5a2d2a6c-a3f3-40fe-a993-c72529144b1a/volumes"
Jul 16 22:30:53.427315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:53.427284    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:30:59.031172 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031085    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031484    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031501    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031516    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031522    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031529    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031534    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031542    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031549    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031616    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031624    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031630    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.031634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.031637    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.035761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.035744    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:30:59.038440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.038417    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:30:59.038606 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.038420    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:30:59.042806 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.042781    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:30:59.089629 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.089597    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"]
Jul 16 22:30:59.089965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.089953    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.090016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.089967    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.090016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.089979    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.090016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.089986    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.090116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.090048    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="289e4e4a-f1f5-4db2-8739-17080a098edf" containerName="revoke-keys"
Jul 16 22:30:59.090116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.090062    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a2d2a6c-a3f3-40fe-a993-c72529144b1a" containerName="revoke-keys"
Jul 16 22:30:59.093234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.093217    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:30:59.102120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.102091    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"]
Jul 16 22:30:59.118122 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.118091    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk\" (UID: \"3e050a2c-fee7-413d-8f03-6f2d950cdf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:30:59.218815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.218775    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r\" (UID: \"262ec790-b8fb-4fe8-972f-2e3efce9fa22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:30:59.219007 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.218861    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk\" (UID: \"3e050a2c-fee7-413d-8f03-6f2d950cdf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:30:59.219522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.219506    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk\" (UID: \"3e050a2c-fee7-413d-8f03-6f2d950cdf54\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:30:59.319360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.319273    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r\" (UID: \"262ec790-b8fb-4fe8-972f-2e3efce9fa22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:30:59.319916 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.319897    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r\" (UID: \"262ec790-b8fb-4fe8-972f-2e3efce9fa22\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:30:59.348596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.348567    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:30:59.405036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.405011    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:30:59.519141 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.519112    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:30:59.521712 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:30:59.521687    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e050a2c_fee7_413d_8f03_6f2d950cdf54.slice/crio-a18f8c9211d76ca219bb7d56182b3e0673ae4e882f152872f862f407e8fc2e85 WatchSource:0}: Error finding container a18f8c9211d76ca219bb7d56182b3e0673ae4e882f152872f862f407e8fc2e85: Status 404 returned error can't find the container with id a18f8c9211d76ca219bb7d56182b3e0673ae4e882f152872f862f407e8fc2e85
Jul 16 22:30:59.569627 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:30:59.569579    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"]
Jul 16 22:30:59.570957 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:30:59.570934    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod262ec790_b8fb_4fe8_972f_2e3efce9fa22.slice/crio-787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901 WatchSource:0}: Error finding container 787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901: Status 404 returned error can't find the container with id 787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901
Jul 16 22:31:00.176170 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.176077    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerID="7de326e79dfc56bc7b01d6a76b7f5c7177136af5b1add0aa182c970d81597070" exitCode=6
Jul 16 22:31:00.176170 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.176157    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerDied","Data":"7de326e79dfc56bc7b01d6a76b7f5c7177136af5b1add0aa182c970d81597070"}
Jul 16 22:31:00.176689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.176219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerStarted","Data":"a18f8c9211d76ca219bb7d56182b3e0673ae4e882f152872f862f407e8fc2e85"}
Jul 16 22:31:00.176689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.176363    2555 scope.go:117] "RemoveContainer" containerID="7de326e79dfc56bc7b01d6a76b7f5c7177136af5b1add0aa182c970d81597070"
Jul 16 22:31:00.177949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.177924    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r" event={"ID":"262ec790-b8fb-4fe8-972f-2e3efce9fa22","Type":"ContainerStarted","Data":"5184e54fcb74debc82e65e97e3223c2b4940e1534994fd2fdbaa486130f47419"}
Jul 16 22:31:00.178058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.177969    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r" event={"ID":"262ec790-b8fb-4fe8-972f-2e3efce9fa22","Type":"ContainerStarted","Data":"787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901"}
Jul 16 22:31:00.209303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:00.209253    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r" podStartSLOduration=1.209235207 podStartE2EDuration="1.209235207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:00.208164686 +0000 UTC m=+2515.045218487" watchObservedRunningTime="2026-07-16 22:31:00.209235207 +0000 UTC m=+2515.046289007"
Jul 16 22:31:01.190067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.189967    2555 generic.go:356] "Generic (PLEG): container finished" podID="262ec790-b8fb-4fe8-972f-2e3efce9fa22" containerID="5184e54fcb74debc82e65e97e3223c2b4940e1534994fd2fdbaa486130f47419" exitCode=0
Jul 16 22:31:01.190067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.190051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r" event={"ID":"262ec790-b8fb-4fe8-972f-2e3efce9fa22","Type":"ContainerDied","Data":"5184e54fcb74debc82e65e97e3223c2b4940e1534994fd2fdbaa486130f47419"}
Jul 16 22:31:01.192005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.191983    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerID="c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980" exitCode=6
Jul 16 22:31:01.192106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.192023    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerDied","Data":"c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980"}
Jul 16 22:31:01.192106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.192058    2555 scope.go:117] "RemoveContainer" containerID="7de326e79dfc56bc7b01d6a76b7f5c7177136af5b1add0aa182c970d81597070"
Jul 16 22:31:01.192286 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:01.192272    2555 scope.go:117] "RemoveContainer" containerID="c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980"
Jul 16 22:31:01.192552 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:01.192529    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk_odh-ai-gateway-infra(3e050a2c-fee7-413d-8f03-6f2d950cdf54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54"
Jul 16 22:31:02.200541 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:02.199967    2555 scope.go:117] "RemoveContainer" containerID="c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980"
Jul 16 22:31:02.200541 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:02.200222    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk_odh-ai-gateway-infra(3e050a2c-fee7-413d-8f03-6f2d950cdf54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54"
Jul 16 22:31:02.337434 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:02.337404    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:31:02.446971 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:02.446936    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca\") pod \"262ec790-b8fb-4fe8-972f-2e3efce9fa22\" (UID: \"262ec790-b8fb-4fe8-972f-2e3efce9fa22\") "
Jul 16 22:31:02.447301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:02.447277    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "262ec790-b8fb-4fe8-972f-2e3efce9fa22" (UID: "262ec790-b8fb-4fe8-972f-2e3efce9fa22"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:02.548308 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:02.548274    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/262ec790-b8fb-4fe8-972f-2e3efce9fa22-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:31:03.205656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:03.205627    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r"
Jul 16 22:31:03.206040 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:03.205666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r" event={"ID":"262ec790-b8fb-4fe8-972f-2e3efce9fa22","Type":"ContainerDied","Data":"787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901"}
Jul 16 22:31:03.206040 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:03.205719    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="787b66b3f3de941493d6a10126e623cabadb019a1ccdaabbc5cc85acea901901"
Jul 16 22:31:10.774139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.774103    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:10.774658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.774639    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262ec790-b8fb-4fe8-972f-2e3efce9fa22" containerName="revoke-keys"
Jul 16 22:31:10.774743 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.774661    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="262ec790-b8fb-4fe8-972f-2e3efce9fa22" containerName="revoke-keys"
Jul 16 22:31:10.774799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.774756    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="262ec790-b8fb-4fe8-972f-2e3efce9fa22" containerName="revoke-keys"
Jul 16 22:31:10.777681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.777660    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:10.787594 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.787510    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:10.922638 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:10.922600    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-wms59\" (UID: \"3eef0051-af3e-42ff-b478-edef203c090a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:11.023243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:11.023206    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-wms59\" (UID: \"3eef0051-af3e-42ff-b478-edef203c090a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:11.023911 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:11.023890    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-wms59\" (UID: \"3eef0051-af3e-42ff-b478-edef203c090a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:11.089375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:11.089343    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:11.243639 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:11.243602    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:11.244259 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:31:11.244230    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3eef0051_af3e_42ff_b478_edef203c090a.slice/crio-1adf919296d814531fee5f5bfc01bbafe4695cffb118969a5ae381a44005c09a WatchSource:0}: Error finding container 1adf919296d814531fee5f5bfc01bbafe4695cffb118969a5ae381a44005c09a: Status 404 returned error can't find the container with id 1adf919296d814531fee5f5bfc01bbafe4695cffb118969a5ae381a44005c09a
Jul 16 22:31:12.254451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:12.254362    2555 generic.go:356] "Generic (PLEG): container finished" podID="3eef0051-af3e-42ff-b478-edef203c090a" containerID="d65a03ab078f276b0ef285ec574f253454e2d9feb21a6507c922ff520b8bebeb" exitCode=6
Jul 16 22:31:12.254825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:12.254458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" event={"ID":"3eef0051-af3e-42ff-b478-edef203c090a","Type":"ContainerDied","Data":"d65a03ab078f276b0ef285ec574f253454e2d9feb21a6507c922ff520b8bebeb"}
Jul 16 22:31:12.254825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:12.254544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" event={"ID":"3eef0051-af3e-42ff-b478-edef203c090a","Type":"ContainerStarted","Data":"1adf919296d814531fee5f5bfc01bbafe4695cffb118969a5ae381a44005c09a"}
Jul 16 22:31:12.254825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:12.254661    2555 scope.go:117] "RemoveContainer" containerID="d65a03ab078f276b0ef285ec574f253454e2d9feb21a6507c922ff520b8bebeb"
Jul 16 22:31:12.297842 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:12.297805    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:31:13.263091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:13.262993    2555 generic.go:356] "Generic (PLEG): container finished" podID="3eef0051-af3e-42ff-b478-edef203c090a" containerID="48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb" exitCode=6
Jul 16 22:31:13.263517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:13.263083    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" event={"ID":"3eef0051-af3e-42ff-b478-edef203c090a","Type":"ContainerDied","Data":"48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb"}
Jul 16 22:31:13.263517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:13.263142    2555 scope.go:117] "RemoveContainer" containerID="d65a03ab078f276b0ef285ec574f253454e2d9feb21a6507c922ff520b8bebeb"
Jul 16 22:31:13.263517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:13.263267    2555 scope.go:117] "RemoveContainer" containerID="48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb"
Jul 16 22:31:13.263671 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:13.263513    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-wms59_odh-ai-gateway-infra(3eef0051-af3e-42ff-b478-edef203c090a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" podUID="3eef0051-af3e-42ff-b478-edef203c090a"
Jul 16 22:31:13.671234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:13.671200    2555 scope.go:117] "RemoveContainer" containerID="c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980"
Jul 16 22:31:14.269816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:14.269728    2555 scope.go:117] "RemoveContainer" containerID="48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb"
Jul 16 22:31:14.270218 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:14.269970    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-wms59_odh-ai-gateway-infra(3eef0051-af3e-42ff-b478-edef203c090a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" podUID="3eef0051-af3e-42ff-b478-edef203c090a"
Jul 16 22:31:14.271399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:14.271377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerStarted","Data":"9a20ae1d928163c79ac4f8b64c2d205943e0cf733159411d4284ad329bea97e5"}
Jul 16 22:31:14.303420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:14.303371    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" podStartSLOduration=15.303356608 podStartE2EDuration="15.303356608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:30:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:31:14.301864119 +0000 UTC m=+2529.138917930" watchObservedRunningTime="2026-07-16 22:31:14.303356608 +0000 UTC m=+2529.140410408"
Jul 16 22:31:15.278732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:15.278701    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerID="9a20ae1d928163c79ac4f8b64c2d205943e0cf733159411d4284ad329bea97e5" exitCode=6
Jul 16 22:31:15.279272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:15.278781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerDied","Data":"9a20ae1d928163c79ac4f8b64c2d205943e0cf733159411d4284ad329bea97e5"}
Jul 16 22:31:15.279272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:15.278844    2555 scope.go:117] "RemoveContainer" containerID="c8e9ab1db3a04120b5e4201dcf9d9acd7423f9be69d5449530123bad7839a980"
Jul 16 22:31:15.279272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:15.279084    2555 scope.go:117] "RemoveContainer" containerID="9a20ae1d928163c79ac4f8b64c2d205943e0cf733159411d4284ad329bea97e5"
Jul 16 22:31:15.279434 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:15.279323    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk_odh-ai-gateway-infra(3e050a2c-fee7-413d-8f03-6f2d950cdf54)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54"
Jul 16 22:31:15.312537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:15.312503    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:31:16.421137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:16.421113    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:31:16.571352 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:16.571251    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca\") pod \"3e050a2c-fee7-413d-8f03-6f2d950cdf54\" (UID: \"3e050a2c-fee7-413d-8f03-6f2d950cdf54\") "
Jul 16 22:31:16.571632 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:16.571611    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3e050a2c-fee7-413d-8f03-6f2d950cdf54" (UID: "3e050a2c-fee7-413d-8f03-6f2d950cdf54"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:16.672052 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:16.672021    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3e050a2c-fee7-413d-8f03-6f2d950cdf54-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:31:17.292874 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.292842    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"
Jul 16 22:31:17.292874 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.292864    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk" event={"ID":"3e050a2c-fee7-413d-8f03-6f2d950cdf54","Type":"ContainerDied","Data":"a18f8c9211d76ca219bb7d56182b3e0673ae4e882f152872f862f407e8fc2e85"}
Jul 16 22:31:17.293115 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.292929    2555 scope.go:117] "RemoveContainer" containerID="9a20ae1d928163c79ac4f8b64c2d205943e0cf733159411d4284ad329bea97e5"
Jul 16 22:31:17.328400 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.328358    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:31:17.339443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.339406    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z4cqk"]
Jul 16 22:31:17.676319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:17.676234    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" path="/var/lib/kubelet/pods/3e050a2c-fee7-413d-8f03-6f2d950cdf54/volumes"
Jul 16 22:31:25.673671 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:25.673643    2555 scope.go:117] "RemoveContainer" containerID="48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb"
Jul 16 22:31:26.344199 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:26.344159    2555 generic.go:356] "Generic (PLEG): container finished" podID="3eef0051-af3e-42ff-b478-edef203c090a" containerID="f2e7cf0dc33d509505bc1fc6f9e420ab5c4d0a27a383cab24c5f9b9e2910074e" exitCode=6
Jul 16 22:31:26.344362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:26.344236    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" event={"ID":"3eef0051-af3e-42ff-b478-edef203c090a","Type":"ContainerDied","Data":"f2e7cf0dc33d509505bc1fc6f9e420ab5c4d0a27a383cab24c5f9b9e2910074e"}
Jul 16 22:31:26.344362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:26.344300    2555 scope.go:117] "RemoveContainer" containerID="48057c93f0e8cdaabda549c2c5ad614b31c3885b3b4d34546c55f620b8db28eb"
Jul 16 22:31:26.344616 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:26.344595    2555 scope.go:117] "RemoveContainer" containerID="f2e7cf0dc33d509505bc1fc6f9e420ab5c4d0a27a383cab24c5f9b9e2910074e"
Jul 16 22:31:26.344871 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:26.344849    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-wms59_odh-ai-gateway-infra(3eef0051-af3e-42ff-b478-edef203c090a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" podUID="3eef0051-af3e-42ff-b478-edef203c090a"
Jul 16 22:31:27.393217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:27.393177    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:27.530342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:27.530310    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:27.567129 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:27.567094    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca\") pod \"3eef0051-af3e-42ff-b478-edef203c090a\" (UID: \"3eef0051-af3e-42ff-b478-edef203c090a\") "
Jul 16 22:31:27.567536 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:27.567517    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3eef0051-af3e-42ff-b478-edef203c090a" (UID: "3eef0051-af3e-42ff-b478-edef203c090a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:27.668114 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:27.668027    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3eef0051-af3e-42ff-b478-edef203c090a-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:31:28.358503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:28.358445    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59" event={"ID":"3eef0051-af3e-42ff-b478-edef203c090a","Type":"ContainerDied","Data":"1adf919296d814531fee5f5bfc01bbafe4695cffb118969a5ae381a44005c09a"}
Jul 16 22:31:28.358503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:28.358491    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"
Jul 16 22:31:28.358788 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:28.358536    2555 scope.go:117] "RemoveContainer" containerID="f2e7cf0dc33d509505bc1fc6f9e420ab5c4d0a27a383cab24c5f9b9e2910074e"
Jul 16 22:31:28.435083 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:28.435039    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:28.465002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:28.464968    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-wms59"]
Jul 16 22:31:29.076938 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.076904    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:29.077237 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077225    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077239    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077247    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077253    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077264    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077270    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077280    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077286    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077293    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077298    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077305    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077309    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077402    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077409    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077416    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:29.077485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.077421    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:29.081634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.081615    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:29.085796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.085776    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:31:29.094601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.094584    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:31:29.167788 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.167758    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:29.178290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.178264    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz\" (UID: \"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:29.278838 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.278797    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz\" (UID: \"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:29.279523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.279503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz\" (UID: \"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:29.391941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.391827    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:29.551969 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.550202    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:29.557356 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:31:29.557330    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa0dd4ec_d5c1_4f93_b727_81ab074f78e3.slice/crio-386741385a550c88f9600c59a5fbfc6067e2e37209dd94975070ad465222b8e5 WatchSource:0}: Error finding container 386741385a550c88f9600c59a5fbfc6067e2e37209dd94975070ad465222b8e5: Status 404 returned error can't find the container with id 386741385a550c88f9600c59a5fbfc6067e2e37209dd94975070ad465222b8e5
Jul 16 22:31:29.676633 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:29.676555    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3eef0051-af3e-42ff-b478-edef203c090a" path="/var/lib/kubelet/pods/3eef0051-af3e-42ff-b478-edef203c090a/volumes"
Jul 16 22:31:30.371570 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:30.371439    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerID="9a78ef5bb32da4252996f787b81eda3b5172831a5365d16c5759db0bbdebbc40" exitCode=6
Jul 16 22:31:30.371570 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:30.371531    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" event={"ID":"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3","Type":"ContainerDied","Data":"9a78ef5bb32da4252996f787b81eda3b5172831a5365d16c5759db0bbdebbc40"}
Jul 16 22:31:30.371805 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:30.371601    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" event={"ID":"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3","Type":"ContainerStarted","Data":"386741385a550c88f9600c59a5fbfc6067e2e37209dd94975070ad465222b8e5"}
Jul 16 22:31:30.371872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:30.371825    2555 scope.go:117] "RemoveContainer" containerID="9a78ef5bb32da4252996f787b81eda3b5172831a5365d16c5759db0bbdebbc40"
Jul 16 22:31:31.378770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:31.378732    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerID="d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b" exitCode=6
Jul 16 22:31:31.379219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:31.378797    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" event={"ID":"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3","Type":"ContainerDied","Data":"d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b"}
Jul 16 22:31:31.379219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:31.378855    2555 scope.go:117] "RemoveContainer" containerID="9a78ef5bb32da4252996f787b81eda3b5172831a5365d16c5759db0bbdebbc40"
Jul 16 22:31:31.379219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:31.379067    2555 scope.go:117] "RemoveContainer" containerID="d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b"
Jul 16 22:31:31.379390 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:31.379253    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz_odh-ai-gateway-infra(fa0dd4ec-d5c1-4f93-b727-81ab074f78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3"
Jul 16 22:31:32.385216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:32.385191    2555 scope.go:117] "RemoveContainer" containerID="d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b"
Jul 16 22:31:32.385618 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:32.385382    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz_odh-ai-gateway-infra(fa0dd4ec-d5c1-4f93-b727-81ab074f78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3"
Jul 16 22:31:43.676273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:43.676247    2555 scope.go:117] "RemoveContainer" containerID="d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b"
Jul 16 22:31:44.446990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:44.446959    2555 generic.go:356] "Generic (PLEG): container finished" podID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerID="b2e30dd7752d3802a189ecfcfe9dde09e12a09d858ea2ec9136f7878a8d1a00e" exitCode=6
Jul 16 22:31:44.447173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:44.447021    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" event={"ID":"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3","Type":"ContainerDied","Data":"b2e30dd7752d3802a189ecfcfe9dde09e12a09d858ea2ec9136f7878a8d1a00e"}
Jul 16 22:31:44.447173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:44.447069    2555 scope.go:117] "RemoveContainer" containerID="d7893bfcc2633701ce76eb0724ec09a82ba69239788661b7b9144e034014992b"
Jul 16 22:31:44.447388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:44.447365    2555 scope.go:117] "RemoveContainer" containerID="b2e30dd7752d3802a189ecfcfe9dde09e12a09d858ea2ec9136f7878a8d1a00e"
Jul 16 22:31:44.447632 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:44.447610    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz_odh-ai-gateway-infra(fa0dd4ec-d5c1-4f93-b727-81ab074f78e3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3"
Jul 16 22:31:45.501972 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:45.501933    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:45.658530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:45.658501    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:45.710536 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:45.710504    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca\") pod \"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3\" (UID: \"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3\") "
Jul 16 22:31:45.710862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:45.710841    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" (UID: "fa0dd4ec-d5c1-4f93-b727-81ab074f78e3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:31:45.811846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:45.811817    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:31:46.461879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:46.461844    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"
Jul 16 22:31:46.462179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:46.461849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz" event={"ID":"fa0dd4ec-d5c1-4f93-b727-81ab074f78e3","Type":"ContainerDied","Data":"386741385a550c88f9600c59a5fbfc6067e2e37209dd94975070ad465222b8e5"}
Jul 16 22:31:46.462179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:46.461990    2555 scope.go:117] "RemoveContainer" containerID="b2e30dd7752d3802a189ecfcfe9dde09e12a09d858ea2ec9136f7878a8d1a00e"
Jul 16 22:31:46.488180 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:46.488153    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:46.493764 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:46.493739    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-sxbjz"]
Jul 16 22:31:47.676323 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:47.676287    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" path="/var/lib/kubelet/pods/fa0dd4ec-d5c1-4f93-b727-81ab074f78e3/volumes"
Jul 16 22:31:49.108773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.108740    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:31:49.109277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109253    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.109358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109280    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.109358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109298    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.109358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109308    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.109543 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109441    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3eef0051-af3e-42ff-b478-edef203c090a" containerName="revoke-keys"
Jul 16 22:31:49.109543 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109456    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.109543 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109491    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e050a2c-fee7-413d-8f03-6f2d950cdf54" containerName="revoke-keys"
Jul 16 22:31:49.109543 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.109499    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:31:49.113433 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.113411    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:31:49.116362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.116340    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:31:49.116680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.116661    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:31:49.126830 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.126807    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:31:49.140289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.140263    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg\" (UID: \"edf2126b-3de2-4542-b225-33b6fc5575a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:31:49.242112 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.242071    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg\" (UID: \"edf2126b-3de2-4542-b225-33b6fc5575a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:31:49.242769 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.242750    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg\" (UID: \"edf2126b-3de2-4542-b225-33b6fc5575a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:31:49.423257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.423169    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:31:49.585661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:49.585628    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:31:49.586933 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:31:49.586913    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedf2126b_3de2_4542_b225_33b6fc5575a0.slice/crio-9d3ad760381c6ebe9ac6ab3b13ead6cfb4d80fa9a8e5a78750c590c16f73e401 WatchSource:0}: Error finding container 9d3ad760381c6ebe9ac6ab3b13ead6cfb4d80fa9a8e5a78750c590c16f73e401: Status 404 returned error can't find the container with id 9d3ad760381c6ebe9ac6ab3b13ead6cfb4d80fa9a8e5a78750c590c16f73e401
Jul 16 22:31:50.487731 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:50.487698    2555 generic.go:356] "Generic (PLEG): container finished" podID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerID="4e319862ab15835fa3a96418e3dd5c4850259eb3fa3a0f08cd44f11546284018" exitCode=6
Jul 16 22:31:50.488241 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:50.487787    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" event={"ID":"edf2126b-3de2-4542-b225-33b6fc5575a0","Type":"ContainerDied","Data":"4e319862ab15835fa3a96418e3dd5c4850259eb3fa3a0f08cd44f11546284018"}
Jul 16 22:31:50.488241 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:50.487858    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" event={"ID":"edf2126b-3de2-4542-b225-33b6fc5575a0","Type":"ContainerStarted","Data":"9d3ad760381c6ebe9ac6ab3b13ead6cfb4d80fa9a8e5a78750c590c16f73e401"}
Jul 16 22:31:50.488241 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:50.488092    2555 scope.go:117] "RemoveContainer" containerID="4e319862ab15835fa3a96418e3dd5c4850259eb3fa3a0f08cd44f11546284018"
Jul 16 22:31:51.495505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:51.495382    2555 generic.go:356] "Generic (PLEG): container finished" podID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerID="71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121" exitCode=6
Jul 16 22:31:51.495505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:51.495441    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" event={"ID":"edf2126b-3de2-4542-b225-33b6fc5575a0","Type":"ContainerDied","Data":"71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121"}
Jul 16 22:31:51.496053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:51.495522    2555 scope.go:117] "RemoveContainer" containerID="4e319862ab15835fa3a96418e3dd5c4850259eb3fa3a0f08cd44f11546284018"
Jul 16 22:31:51.496053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:51.495701    2555 scope.go:117] "RemoveContainer" containerID="71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121"
Jul 16 22:31:51.496053 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:51.495957    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg_odh-ai-gateway-infra(edf2126b-3de2-4542-b225-33b6fc5575a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0"
Jul 16 22:31:52.502346 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:31:52.502314    2555 scope.go:117] "RemoveContainer" containerID="71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121"
Jul 16 22:31:52.502769 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:31:52.502546    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg_odh-ai-gateway-infra(edf2126b-3de2-4542-b225-33b6fc5575a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0"
Jul 16 22:32:00.051726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.051686    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:32:00.888389 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.888352    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:00.888734 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.888720    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:32:00.888780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.888735    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:32:00.888815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.888797    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa0dd4ec-d5c1-4f93-b727-81ab074f78e3" containerName="revoke-keys"
Jul 16 22:32:00.893533 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.893510    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:00.908506 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.908455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:00.946073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:00.946022    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ddd6b\" (UID: \"a84facbe-0497-413e-975b-adea9503c7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:01.047376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:01.047338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ddd6b\" (UID: \"a84facbe-0497-413e-975b-adea9503c7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:01.048023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:01.047997    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ddd6b\" (UID: \"a84facbe-0497-413e-975b-adea9503c7f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:01.211213 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:01.211128    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:01.364171 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:01.364146    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:01.366223 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:32:01.366196    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda84facbe_0497_413e_975b_adea9503c7f4.slice/crio-a0b703d9a71a0e15ee20415148eaf94a57fc352eb0d58908ce3578ea5e1adc64 WatchSource:0}: Error finding container a0b703d9a71a0e15ee20415148eaf94a57fc352eb0d58908ce3578ea5e1adc64: Status 404 returned error can't find the container with id a0b703d9a71a0e15ee20415148eaf94a57fc352eb0d58908ce3578ea5e1adc64
Jul 16 22:32:01.550333 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:01.550293    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" event={"ID":"a84facbe-0497-413e-975b-adea9503c7f4","Type":"ContainerStarted","Data":"a0b703d9a71a0e15ee20415148eaf94a57fc352eb0d58908ce3578ea5e1adc64"}
Jul 16 22:32:02.557102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:02.557061    2555 generic.go:356] "Generic (PLEG): container finished" podID="a84facbe-0497-413e-975b-adea9503c7f4" containerID="9768d16fe58dca328f35ce52a25e46ead738889a0d453270f915c42758012c9c" exitCode=6
Jul 16 22:32:02.557514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:02.557140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" event={"ID":"a84facbe-0497-413e-975b-adea9503c7f4","Type":"ContainerDied","Data":"9768d16fe58dca328f35ce52a25e46ead738889a0d453270f915c42758012c9c"}
Jul 16 22:32:02.557514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:02.557341    2555 scope.go:117] "RemoveContainer" containerID="9768d16fe58dca328f35ce52a25e46ead738889a0d453270f915c42758012c9c"
Jul 16 22:32:03.564352 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:03.564319    2555 generic.go:356] "Generic (PLEG): container finished" podID="a84facbe-0497-413e-975b-adea9503c7f4" containerID="e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd" exitCode=6
Jul 16 22:32:03.564862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:03.564351    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" event={"ID":"a84facbe-0497-413e-975b-adea9503c7f4","Type":"ContainerDied","Data":"e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd"}
Jul 16 22:32:03.564862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:03.564417    2555 scope.go:117] "RemoveContainer" containerID="9768d16fe58dca328f35ce52a25e46ead738889a0d453270f915c42758012c9c"
Jul 16 22:32:03.564862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:03.564668    2555 scope.go:117] "RemoveContainer" containerID="e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd"
Jul 16 22:32:03.565012 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:03.564930    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ddd6b_odh-ai-gateway-infra(a84facbe-0497-413e-975b-adea9503c7f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" podUID="a84facbe-0497-413e-975b-adea9503c7f4"
Jul 16 22:32:04.571505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:04.571480    2555 scope.go:117] "RemoveContainer" containerID="e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd"
Jul 16 22:32:04.571890 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:04.571678    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ddd6b_odh-ai-gateway-infra(a84facbe-0497-413e-975b-adea9503c7f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" podUID="a84facbe-0497-413e-975b-adea9503c7f4"
Jul 16 22:32:05.678716 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:05.678687    2555 scope.go:117] "RemoveContainer" containerID="71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121"
Jul 16 22:32:06.583690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:06.583594    2555 generic.go:356] "Generic (PLEG): container finished" podID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerID="8afcf5c164eb32a94f2e09185eb618b08422b288a45ede44e15f2dda3d710a98" exitCode=6
Jul 16 22:32:06.583690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:06.583671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" event={"ID":"edf2126b-3de2-4542-b225-33b6fc5575a0","Type":"ContainerDied","Data":"8afcf5c164eb32a94f2e09185eb618b08422b288a45ede44e15f2dda3d710a98"}
Jul 16 22:32:06.583927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:06.583744    2555 scope.go:117] "RemoveContainer" containerID="71c365838e429850bd11e6be2bcde3bda72aece96cf41bea398bf4a5b7f19121"
Jul 16 22:32:06.584118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:06.584092    2555 scope.go:117] "RemoveContainer" containerID="8afcf5c164eb32a94f2e09185eb618b08422b288a45ede44e15f2dda3d710a98"
Jul 16 22:32:06.584336 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:06.584314    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg_odh-ai-gateway-infra(edf2126b-3de2-4542-b225-33b6fc5575a0)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0"
Jul 16 22:32:07.643858 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:07.643822    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:32:07.777668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:07.777645    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:32:07.907527 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:07.907438    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca\") pod \"edf2126b-3de2-4542-b225-33b6fc5575a0\" (UID: \"edf2126b-3de2-4542-b225-33b6fc5575a0\") "
Jul 16 22:32:07.907852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:07.907831    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "edf2126b-3de2-4542-b225-33b6fc5575a0" (UID: "edf2126b-3de2-4542-b225-33b6fc5575a0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:08.008964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.008925    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/edf2126b-3de2-4542-b225-33b6fc5575a0-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:08.600739 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.600695    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"
Jul 16 22:32:08.600889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.600733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg" event={"ID":"edf2126b-3de2-4542-b225-33b6fc5575a0","Type":"ContainerDied","Data":"9d3ad760381c6ebe9ac6ab3b13ead6cfb4d80fa9a8e5a78750c590c16f73e401"}
Jul 16 22:32:08.600889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.600802    2555 scope.go:117] "RemoveContainer" containerID="8afcf5c164eb32a94f2e09185eb618b08422b288a45ede44e15f2dda3d710a98"
Jul 16 22:32:08.633275 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.633244    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:32:08.641341 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.641311    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-tq5jg"]
Jul 16 22:32:08.990385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:08.990297    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:32:09.675827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:09.675793    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" path="/var/lib/kubelet/pods/edf2126b-3de2-4542-b225-33b6fc5575a0/volumes"
Jul 16 22:32:10.307362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307328    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:10.307752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307694    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.307752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307706    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.307752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307717    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.307752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307723    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.307881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307777    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.307881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.307787    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:10.311935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.311917    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:10.319253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.319229    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:10.429902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.429869    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt\" (UID: \"cfa78e8f-aedd-4d9f-917f-1d299534fd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:10.530987 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.530948    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt\" (UID: \"cfa78e8f-aedd-4d9f-917f-1d299534fd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:10.531662 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.531638    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt\" (UID: \"cfa78e8f-aedd-4d9f-917f-1d299534fd81\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:10.623444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.623348    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:10.764898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:10.764846    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:10.767122 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:32:10.767091    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfa78e8f_aedd_4d9f_917f_1d299534fd81.slice/crio-b396216145b74ed9696d99aca8266f56f8c595cc06ca33161902d9a81aba0ce0 WatchSource:0}: Error finding container b396216145b74ed9696d99aca8266f56f8c595cc06ca33161902d9a81aba0ce0: Status 404 returned error can't find the container with id b396216145b74ed9696d99aca8266f56f8c595cc06ca33161902d9a81aba0ce0
Jul 16 22:32:11.619012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:11.618967    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerID="5bd77059451c5eadfde4be99e5feff6e19c1d4bc65166bb469a86964c2e1ff57" exitCode=6
Jul 16 22:32:11.619443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:11.619051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" event={"ID":"cfa78e8f-aedd-4d9f-917f-1d299534fd81","Type":"ContainerDied","Data":"5bd77059451c5eadfde4be99e5feff6e19c1d4bc65166bb469a86964c2e1ff57"}
Jul 16 22:32:11.619443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:11.619104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" event={"ID":"cfa78e8f-aedd-4d9f-917f-1d299534fd81","Type":"ContainerStarted","Data":"b396216145b74ed9696d99aca8266f56f8c595cc06ca33161902d9a81aba0ce0"}
Jul 16 22:32:11.619443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:11.619209    2555 scope.go:117] "RemoveContainer" containerID="5bd77059451c5eadfde4be99e5feff6e19c1d4bc65166bb469a86964c2e1ff57"
Jul 16 22:32:12.626448 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:12.626364    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerID="d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71" exitCode=6
Jul 16 22:32:12.626828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:12.626455    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" event={"ID":"cfa78e8f-aedd-4d9f-917f-1d299534fd81","Type":"ContainerDied","Data":"d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71"}
Jul 16 22:32:12.626828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:12.626527    2555 scope.go:117] "RemoveContainer" containerID="5bd77059451c5eadfde4be99e5feff6e19c1d4bc65166bb469a86964c2e1ff57"
Jul 16 22:32:12.626828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:12.626664    2555 scope.go:117] "RemoveContainer" containerID="d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71"
Jul 16 22:32:12.626984 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:12.626895    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt_odh-ai-gateway-infra(cfa78e8f-aedd-4d9f-917f-1d299534fd81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81"
Jul 16 22:32:13.633231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:13.633200    2555 scope.go:117] "RemoveContainer" containerID="d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71"
Jul 16 22:32:13.633643 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:13.633404    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt_odh-ai-gateway-infra(cfa78e8f-aedd-4d9f-917f-1d299534fd81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81"
Jul 16 22:32:17.670852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:17.670817    2555 scope.go:117] "RemoveContainer" containerID="e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd"
Jul 16 22:32:18.661429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:18.661346    2555 generic.go:356] "Generic (PLEG): container finished" podID="a84facbe-0497-413e-975b-adea9503c7f4" containerID="d22758b0bc7d50f461b2dd96b8a4fae3da430dcf712328fdf04fdc25e212dace" exitCode=6
Jul 16 22:32:18.661429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:18.661392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" event={"ID":"a84facbe-0497-413e-975b-adea9503c7f4","Type":"ContainerDied","Data":"d22758b0bc7d50f461b2dd96b8a4fae3da430dcf712328fdf04fdc25e212dace"}
Jul 16 22:32:18.661637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:18.661450    2555 scope.go:117] "RemoveContainer" containerID="e68cb774145b6bd205300d718254d57c59d3708646f7ceebb2a330533d28aacd"
Jul 16 22:32:18.661804 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:18.661780    2555 scope.go:117] "RemoveContainer" containerID="d22758b0bc7d50f461b2dd96b8a4fae3da430dcf712328fdf04fdc25e212dace"
Jul 16 22:32:18.662040 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:18.662017    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ddd6b_odh-ai-gateway-infra(a84facbe-0497-413e-975b-adea9503c7f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" podUID="a84facbe-0497-413e-975b-adea9503c7f4"
Jul 16 22:32:19.014261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.009069    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:19.014261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.009923    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:19.014261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.009944    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:19.014261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.010131    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="edf2126b-3de2-4542-b225-33b6fc5575a0" containerName="revoke-keys"
Jul 16 22:32:19.015022 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.015003    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:19.023862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.023837    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:19.106623 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.106578    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2\" (UID: \"28dd973c-156e-4174-924a-9d3a19384b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:19.207264 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.207219    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2\" (UID: \"28dd973c-156e-4174-924a-9d3a19384b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:19.207882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.207861    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2\" (UID: \"28dd973c-156e-4174-924a-9d3a19384b34\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:19.326964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.326929    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:19.468779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.468748    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:19.470617 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:32:19.470590    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28dd973c_156e_4174_924a_9d3a19384b34.slice/crio-87a363fc82ef6b9399f7c2fde1457bf44fe24af92df7dabbdcb869ea8d4b78c0 WatchSource:0}: Error finding container 87a363fc82ef6b9399f7c2fde1457bf44fe24af92df7dabbdcb869ea8d4b78c0: Status 404 returned error can't find the container with id 87a363fc82ef6b9399f7c2fde1457bf44fe24af92df7dabbdcb869ea8d4b78c0
Jul 16 22:32:19.667994 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.667896    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" event={"ID":"28dd973c-156e-4174-924a-9d3a19384b34","Type":"ContainerStarted","Data":"87a363fc82ef6b9399f7c2fde1457bf44fe24af92df7dabbdcb869ea8d4b78c0"}
Jul 16 22:32:19.692307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.692277    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:19.820345 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.820325    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:19.914394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.914363    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca\") pod \"a84facbe-0497-413e-975b-adea9503c7f4\" (UID: \"a84facbe-0497-413e-975b-adea9503c7f4\") "
Jul 16 22:32:19.914784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:19.914760    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a84facbe-0497-413e-975b-adea9503c7f4" (UID: "a84facbe-0497-413e-975b-adea9503c7f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:20.015324 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.015237    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a84facbe-0497-413e-975b-adea9503c7f4-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:20.676598 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.676524    2555 generic.go:356] "Generic (PLEG): container finished" podID="28dd973c-156e-4174-924a-9d3a19384b34" containerID="a65268c730e6a26745b78f74bbcc7e1b801dea4dc52e340a2aa8af851ae08594" exitCode=6
Jul 16 22:32:20.676754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.676611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" event={"ID":"28dd973c-156e-4174-924a-9d3a19384b34","Type":"ContainerDied","Data":"a65268c730e6a26745b78f74bbcc7e1b801dea4dc52e340a2aa8af851ae08594"}
Jul 16 22:32:20.676754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.676708    2555 scope.go:117] "RemoveContainer" containerID="a65268c730e6a26745b78f74bbcc7e1b801dea4dc52e340a2aa8af851ae08594"
Jul 16 22:32:20.677921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.677892    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"
Jul 16 22:32:20.678026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.677932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b" event={"ID":"a84facbe-0497-413e-975b-adea9503c7f4","Type":"ContainerDied","Data":"a0b703d9a71a0e15ee20415148eaf94a57fc352eb0d58908ce3578ea5e1adc64"}
Jul 16 22:32:20.678026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.677976    2555 scope.go:117] "RemoveContainer" containerID="d22758b0bc7d50f461b2dd96b8a4fae3da430dcf712328fdf04fdc25e212dace"
Jul 16 22:32:20.724984 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.724959    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:20.729679 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:20.729657    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ddd6b"]
Jul 16 22:32:21.677147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:21.677071    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a84facbe-0497-413e-975b-adea9503c7f4" path="/var/lib/kubelet/pods/a84facbe-0497-413e-975b-adea9503c7f4/volumes"
Jul 16 22:32:21.683681 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:21.683654    2555 generic.go:356] "Generic (PLEG): container finished" podID="28dd973c-156e-4174-924a-9d3a19384b34" containerID="b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f" exitCode=6
Jul 16 22:32:21.683794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:21.683731    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" event={"ID":"28dd973c-156e-4174-924a-9d3a19384b34","Type":"ContainerDied","Data":"b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f"}
Jul 16 22:32:21.683794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:21.683781    2555 scope.go:117] "RemoveContainer" containerID="a65268c730e6a26745b78f74bbcc7e1b801dea4dc52e340a2aa8af851ae08594"
Jul 16 22:32:21.683947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:21.683932    2555 scope.go:117] "RemoveContainer" containerID="b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f"
Jul 16 22:32:21.684180 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:21.684157    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2_odh-ai-gateway-infra(28dd973c-156e-4174-924a-9d3a19384b34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" podUID="28dd973c-156e-4174-924a-9d3a19384b34"
Jul 16 22:32:22.691512 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:22.691453    2555 scope.go:117] "RemoveContainer" containerID="b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f"
Jul 16 22:32:22.691900 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:22.691676    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2_odh-ai-gateway-infra(28dd973c-156e-4174-924a-9d3a19384b34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" podUID="28dd973c-156e-4174-924a-9d3a19384b34"
Jul 16 22:32:24.134729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:24.134692    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:32:26.670994 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:26.670918    2555 scope.go:117] "RemoveContainer" containerID="d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71"
Jul 16 22:32:27.320336 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320308    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"]
Jul 16 22:32:27.320752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320736    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320759    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320773    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320816 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320781    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320888    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320900    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.320917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.320905    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:27.323722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.323707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:27.332263 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.332242    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"]
Jul 16 22:32:27.383366 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.383337    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq\" (UID: \"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:27.484022 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.483985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq\" (UID: \"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:27.484675 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.484655    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq\" (UID: \"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:27.682011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.681935    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:27.717524 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.717488    2555 generic.go:356] "Generic (PLEG): container finished" podID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerID="c50c38cb8325339b78b0cea332f4d96b6c4179e8d1bdb81f0099c69d348fff50" exitCode=6
Jul 16 22:32:27.717524 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.717503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" event={"ID":"cfa78e8f-aedd-4d9f-917f-1d299534fd81","Type":"ContainerDied","Data":"c50c38cb8325339b78b0cea332f4d96b6c4179e8d1bdb81f0099c69d348fff50"}
Jul 16 22:32:27.717735 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.717566    2555 scope.go:117] "RemoveContainer" containerID="d4ef31ba7f004723c3feba2a4c294dc260f46a4063c02e1aa3017149e552ff71"
Jul 16 22:32:27.717931 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.717914    2555 scope.go:117] "RemoveContainer" containerID="c50c38cb8325339b78b0cea332f4d96b6c4179e8d1bdb81f0099c69d348fff50"
Jul 16 22:32:27.718148 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:27.718122    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt_odh-ai-gateway-infra(cfa78e8f-aedd-4d9f-917f-1d299534fd81)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81"
Jul 16 22:32:27.827795 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:27.827773    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"]
Jul 16 22:32:27.829322 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:32:27.829289    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5dd0a35_fdfd_44f3_92c5_a0e699e2e495.slice/crio-56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8 WatchSource:0}: Error finding container 56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8: Status 404 returned error can't find the container with id 56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8
Jul 16 22:32:28.725014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.724979    2555 generic.go:356] "Generic (PLEG): container finished" podID="e5dd0a35-fdfd-44f3-92c5-a0e699e2e495" containerID="ec79f3310e71afde2fdd6f12a2bf98f43f21ee01d404813a6ef904e2892c8f38" exitCode=0
Jul 16 22:32:28.725505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.725074    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq" event={"ID":"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495","Type":"ContainerDied","Data":"ec79f3310e71afde2fdd6f12a2bf98f43f21ee01d404813a6ef904e2892c8f38"}
Jul 16 22:32:28.725505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.725162    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq" event={"ID":"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495","Type":"ContainerStarted","Data":"56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8"}
Jul 16 22:32:28.752100 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.751999    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:28.881775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.881750    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:28.894896 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.894868    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca\") pod \"cfa78e8f-aedd-4d9f-917f-1d299534fd81\" (UID: \"cfa78e8f-aedd-4d9f-917f-1d299534fd81\") "
Jul 16 22:32:28.895193 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.895173    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cfa78e8f-aedd-4d9f-917f-1d299534fd81" (UID: "cfa78e8f-aedd-4d9f-917f-1d299534fd81"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:28.995882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:28.995845    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cfa78e8f-aedd-4d9f-917f-1d299534fd81-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:29.733013 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.732982    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"
Jul 16 22:32:29.733491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.733029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt" event={"ID":"cfa78e8f-aedd-4d9f-917f-1d299534fd81","Type":"ContainerDied","Data":"b396216145b74ed9696d99aca8266f56f8c595cc06ca33161902d9a81aba0ce0"}
Jul 16 22:32:29.733491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.733091    2555 scope.go:117] "RemoveContainer" containerID="c50c38cb8325339b78b0cea332f4d96b6c4179e8d1bdb81f0099c69d348fff50"
Jul 16 22:32:29.763554 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.763518    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:29.773551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.773519    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-7tvpt"]
Jul 16 22:32:29.866329 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.866301    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:29.902126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.902096    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca\") pod \"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495\" (UID: \"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495\") "
Jul 16 22:32:29.902424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:29.902403    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e5dd0a35-fdfd-44f3-92c5-a0e699e2e495" (UID: "e5dd0a35-fdfd-44f3-92c5-a0e699e2e495"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:30.002780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:30.002679    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e5dd0a35-fdfd-44f3-92c5-a0e699e2e495-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:30.740703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:30.740669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq" event={"ID":"e5dd0a35-fdfd-44f3-92c5-a0e699e2e495","Type":"ContainerDied","Data":"56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8"}
Jul 16 22:32:30.740703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:30.740704    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq"
Jul 16 22:32:30.741145 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:30.740718    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56fee37df67138340f94c054b8e2082738f95bca1d4c7b1cadb0a50cbf8671f8"
Jul 16 22:32:31.675868 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:31.675837    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" path="/var/lib/kubelet/pods/cfa78e8f-aedd-4d9f-917f-1d299534fd81/volumes"
Jul 16 22:32:35.674012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:35.673982    2555 scope.go:117] "RemoveContainer" containerID="b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f"
Jul 16 22:32:36.777674 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:36.777646    2555 generic.go:356] "Generic (PLEG): container finished" podID="28dd973c-156e-4174-924a-9d3a19384b34" containerID="58784045a84f3369f1238cc0c56591c64fed88a74d376aa33bb5fda8140032f5" exitCode=6
Jul 16 22:32:36.778107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:36.777725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" event={"ID":"28dd973c-156e-4174-924a-9d3a19384b34","Type":"ContainerDied","Data":"58784045a84f3369f1238cc0c56591c64fed88a74d376aa33bb5fda8140032f5"}
Jul 16 22:32:36.778107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:36.777804    2555 scope.go:117] "RemoveContainer" containerID="b9daa697f680387e7e6eb08617c057bb4c7f87e9c590e22a2cf988a01c1cba8f"
Jul 16 22:32:36.778208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:36.778125    2555 scope.go:117] "RemoveContainer" containerID="58784045a84f3369f1238cc0c56591c64fed88a74d376aa33bb5fda8140032f5"
Jul 16 22:32:36.778367 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:36.778339    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2_odh-ai-gateway-infra(28dd973c-156e-4174-924a-9d3a19384b34)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" podUID="28dd973c-156e-4174-924a-9d3a19384b34"
Jul 16 22:32:37.820216 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:37.820182    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:37.960544 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:37.960521    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:37.966396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:37.966362    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca\") pod \"28dd973c-156e-4174-924a-9d3a19384b34\" (UID: \"28dd973c-156e-4174-924a-9d3a19384b34\") "
Jul 16 22:32:37.966723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:37.966701    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "28dd973c-156e-4174-924a-9d3a19384b34" (UID: "28dd973c-156e-4174-924a-9d3a19384b34"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:38.067447 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.067388    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/28dd973c-156e-4174-924a-9d3a19384b34-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:38.792517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.792483    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2" event={"ID":"28dd973c-156e-4174-924a-9d3a19384b34","Type":"ContainerDied","Data":"87a363fc82ef6b9399f7c2fde1457bf44fe24af92df7dabbdcb869ea8d4b78c0"}
Jul 16 22:32:38.792690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.792546    2555 scope.go:117] "RemoveContainer" containerID="58784045a84f3369f1238cc0c56591c64fed88a74d376aa33bb5fda8140032f5"
Jul 16 22:32:38.792690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.792492    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"
Jul 16 22:32:38.820728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.820703    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:38.824372 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:38.824352    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-8xkp2"]
Jul 16 22:32:39.145095 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145019    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:39.145484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145433    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145451    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145476    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145483    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145492    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145501    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145511    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e5dd0a35-fdfd-44f3-92c5-a0e699e2e495" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145516    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e5dd0a35-fdfd-44f3-92c5-a0e699e2e495" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145527    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145531    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a84facbe-0497-413e-975b-adea9503c7f4" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145538    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145543    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145606    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145613    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145621    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145626    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145632    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e5dd0a35-fdfd-44f3-92c5-a0e699e2e495" containerName="revoke-keys"
Jul 16 22:32:39.145740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.145638    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:32:39.149935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.149919    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:39.153073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.153050    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:32:39.153176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.153100    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:32:39.157161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.157138    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:39.176793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.176766    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j\" (UID: \"d814edab-01a6-4a8d-be2a-6aa4745b3a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:39.277628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.277589    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j\" (UID: \"d814edab-01a6-4a8d-be2a-6aa4745b3a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:39.278192 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.278172    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j\" (UID: \"d814edab-01a6-4a8d-be2a-6aa4745b3a75\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:39.462621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.462529    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:39.609589 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.609562    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:39.611410 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:32:39.611378    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd814edab_01a6_4a8d_be2a_6aa4745b3a75.slice/crio-797b8f2b6996d448ff216bb1f44616615e5f50646ae038a28d49b08edbf5a3e4 WatchSource:0}: Error finding container 797b8f2b6996d448ff216bb1f44616615e5f50646ae038a28d49b08edbf5a3e4: Status 404 returned error can't find the container with id 797b8f2b6996d448ff216bb1f44616615e5f50646ae038a28d49b08edbf5a3e4
Jul 16 22:32:39.676652 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.676622    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28dd973c-156e-4174-924a-9d3a19384b34" path="/var/lib/kubelet/pods/28dd973c-156e-4174-924a-9d3a19384b34/volumes"
Jul 16 22:32:39.798830 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:39.798798    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" event={"ID":"d814edab-01a6-4a8d-be2a-6aa4745b3a75","Type":"ContainerStarted","Data":"797b8f2b6996d448ff216bb1f44616615e5f50646ae038a28d49b08edbf5a3e4"}
Jul 16 22:32:40.806940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:40.806908    2555 generic.go:356] "Generic (PLEG): container finished" podID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerID="9bdc8734f107fe570bb54a0d85d8bc3fd7354430fe2bae16914ca5eb11b91b94" exitCode=6
Jul 16 22:32:40.807346 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:40.807008    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" event={"ID":"d814edab-01a6-4a8d-be2a-6aa4745b3a75","Type":"ContainerDied","Data":"9bdc8734f107fe570bb54a0d85d8bc3fd7354430fe2bae16914ca5eb11b91b94"}
Jul 16 22:32:40.807346 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:40.807147    2555 scope.go:117] "RemoveContainer" containerID="9bdc8734f107fe570bb54a0d85d8bc3fd7354430fe2bae16914ca5eb11b91b94"
Jul 16 22:32:41.536337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:41.536310    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:32:41.814603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:41.814570    2555 generic.go:356] "Generic (PLEG): container finished" podID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerID="63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7" exitCode=6
Jul 16 22:32:41.815020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:41.814656    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" event={"ID":"d814edab-01a6-4a8d-be2a-6aa4745b3a75","Type":"ContainerDied","Data":"63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7"}
Jul 16 22:32:41.815020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:41.814714    2555 scope.go:117] "RemoveContainer" containerID="9bdc8734f107fe570bb54a0d85d8bc3fd7354430fe2bae16914ca5eb11b91b94"
Jul 16 22:32:41.815020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:41.814842    2555 scope.go:117] "RemoveContainer" containerID="63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7"
Jul 16 22:32:41.815184 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:41.815073    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j_odh-ai-gateway-infra(d814edab-01a6-4a8d-be2a-6aa4745b3a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75"
Jul 16 22:32:42.822266 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:42.822242    2555 scope.go:117] "RemoveContainer" containerID="63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7"
Jul 16 22:32:42.822683 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:42.822433    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j_odh-ai-gateway-infra(d814edab-01a6-4a8d-be2a-6aa4745b3a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75"
Jul 16 22:32:54.671300 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:54.671269    2555 scope.go:117] "RemoveContainer" containerID="63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7"
Jul 16 22:32:55.893797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:55.893766    2555 generic.go:356] "Generic (PLEG): container finished" podID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerID="725070a16a33ccb4daf372b4149a8ca9a2c8eb2b1c714a76781050c980e1d667" exitCode=6
Jul 16 22:32:55.894274 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:55.893837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" event={"ID":"d814edab-01a6-4a8d-be2a-6aa4745b3a75","Type":"ContainerDied","Data":"725070a16a33ccb4daf372b4149a8ca9a2c8eb2b1c714a76781050c980e1d667"}
Jul 16 22:32:55.894274 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:55.893897    2555 scope.go:117] "RemoveContainer" containerID="63da1d3fe1bc4a510dd8aa51302a09d294384ca7670871fc162c599b65a5b1a7"
Jul 16 22:32:55.894410 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:55.894295    2555 scope.go:117] "RemoveContainer" containerID="725070a16a33ccb4daf372b4149a8ca9a2c8eb2b1c714a76781050c980e1d667"
Jul 16 22:32:55.894568 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:32:55.894541    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j_odh-ai-gateway-infra(d814edab-01a6-4a8d-be2a-6aa4745b3a75)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75"
Jul 16 22:32:56.939285 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:56.939252    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:57.077085 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.077062    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:57.139956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.139920    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca\") pod \"d814edab-01a6-4a8d-be2a-6aa4745b3a75\" (UID: \"d814edab-01a6-4a8d-be2a-6aa4745b3a75\") "
Jul 16 22:32:57.140361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.140334    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d814edab-01a6-4a8d-be2a-6aa4745b3a75" (UID: "d814edab-01a6-4a8d-be2a-6aa4745b3a75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:57.240591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.240502    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d814edab-01a6-4a8d-be2a-6aa4745b3a75-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:32:57.907534 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.907503    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"
Jul 16 22:32:57.907738 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.907535    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j" event={"ID":"d814edab-01a6-4a8d-be2a-6aa4745b3a75","Type":"ContainerDied","Data":"797b8f2b6996d448ff216bb1f44616615e5f50646ae038a28d49b08edbf5a3e4"}
Jul 16 22:32:57.907738 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.907596    2555 scope.go:117] "RemoveContainer" containerID="725070a16a33ccb4daf372b4149a8ca9a2c8eb2b1c714a76781050c980e1d667"
Jul 16 22:32:57.930315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.930275    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:57.934756 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:57.934703    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-v9k9j"]
Jul 16 22:32:59.677039 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:32:59.677007    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" path="/var/lib/kubelet/pods/d814edab-01a6-4a8d-be2a-6aa4745b3a75/volumes"
Jul 16 22:33:00.371282 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371247    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:00.371617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371605    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371619    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="cfa78e8f-aedd-4d9f-917f-1d299534fd81" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371628    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371634    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371646    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371653    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371662    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371667    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371677    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371682    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371743    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371750    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371755    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d814edab-01a6-4a8d-be2a-6aa4745b3a75" containerName="revoke-keys"
Jul 16 22:33:00.371882 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.371761    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="28dd973c-156e-4174-924a-9d3a19384b34" containerName="revoke-keys"
Jul 16 22:33:00.376242 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.376217    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:00.383034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.383006    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:00.383241 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.383218    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:33:00.383340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.383183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:00.472043 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.472008    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5\" (UID: \"fd752372-153d-41d0-a275-f8cbc18a5f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:00.573041 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.573005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5\" (UID: \"fd752372-153d-41d0-a275-f8cbc18a5f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:00.573682 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.573664    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5\" (UID: \"fd752372-153d-41d0-a275-f8cbc18a5f41\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:00.689486 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.689357    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:00.832120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.832089    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:00.834914 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:33:00.834887    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd752372_153d_41d0_a275_f8cbc18a5f41.slice/crio-3c60013ba16b33286e441e382c25cf8b8f91956fb66e478f6371ca0a22cc4ea7 WatchSource:0}: Error finding container 3c60013ba16b33286e441e382c25cf8b8f91956fb66e478f6371ca0a22cc4ea7: Status 404 returned error can't find the container with id 3c60013ba16b33286e441e382c25cf8b8f91956fb66e478f6371ca0a22cc4ea7
Jul 16 22:33:00.927051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:00.927015    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" event={"ID":"fd752372-153d-41d0-a275-f8cbc18a5f41","Type":"ContainerStarted","Data":"3c60013ba16b33286e441e382c25cf8b8f91956fb66e478f6371ca0a22cc4ea7"}
Jul 16 22:33:01.933748 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:01.933711    2555 generic.go:356] "Generic (PLEG): container finished" podID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerID="0676f540d7cc75efd4333e651dd77bcaed92f5944fe04bcd6767dd24c66ff10a" exitCode=6
Jul 16 22:33:01.934205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:01.933773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" event={"ID":"fd752372-153d-41d0-a275-f8cbc18a5f41","Type":"ContainerDied","Data":"0676f540d7cc75efd4333e651dd77bcaed92f5944fe04bcd6767dd24c66ff10a"}
Jul 16 22:33:01.934205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:01.933974    2555 scope.go:117] "RemoveContainer" containerID="0676f540d7cc75efd4333e651dd77bcaed92f5944fe04bcd6767dd24c66ff10a"
Jul 16 22:33:02.941230 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:02.941140    2555 generic.go:356] "Generic (PLEG): container finished" podID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerID="b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01" exitCode=6
Jul 16 22:33:02.941230 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:02.941184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" event={"ID":"fd752372-153d-41d0-a275-f8cbc18a5f41","Type":"ContainerDied","Data":"b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01"}
Jul 16 22:33:02.941684 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:02.941241    2555 scope.go:117] "RemoveContainer" containerID="0676f540d7cc75efd4333e651dd77bcaed92f5944fe04bcd6767dd24c66ff10a"
Jul 16 22:33:02.941684 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:02.941427    2555 scope.go:117] "RemoveContainer" containerID="b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01"
Jul 16 22:33:02.941684 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:02.941654    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5_odh-ai-gateway-infra(fd752372-153d-41d0-a275-f8cbc18a5f41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41"
Jul 16 22:33:03.948265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:03.948238    2555 scope.go:117] "RemoveContainer" containerID="b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01"
Jul 16 22:33:03.948672 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:03.948421    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5_odh-ai-gateway-infra(fd752372-153d-41d0-a275-f8cbc18a5f41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41"
Jul 16 22:33:09.074393 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.074354    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:09.079199 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.079180    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:09.087110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.087021    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:09.249893 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.249858    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4\" (UID: \"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:09.351207 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.351112    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4\" (UID: \"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:09.351761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.351741    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4\" (UID: \"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:09.392123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.392093    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:09.532740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.532708    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:09.533882 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:33:09.533850    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0efbe6e6_fed7_4fe0_a40d_bfd20164aef3.slice/crio-640a1f09b7cbc1ed1ff1c5d7628f0cb2b8ac2ccba8ac0537b88a178bdad01821 WatchSource:0}: Error finding container 640a1f09b7cbc1ed1ff1c5d7628f0cb2b8ac2ccba8ac0537b88a178bdad01821: Status 404 returned error can't find the container with id 640a1f09b7cbc1ed1ff1c5d7628f0cb2b8ac2ccba8ac0537b88a178bdad01821
Jul 16 22:33:09.982201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.982109    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerStarted","Data":"e066ea3e16f358f513df0463a5faba59986d8c79292abc6afafebd42bb0b9559"}
Jul 16 22:33:09.982201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:09.982179    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerStarted","Data":"640a1f09b7cbc1ed1ff1c5d7628f0cb2b8ac2ccba8ac0537b88a178bdad01821"}
Jul 16 22:33:10.000927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:10.000871    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" podStartSLOduration=1.000856942 podStartE2EDuration="1.000856942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:09.998881491 +0000 UTC m=+2644.835935292" watchObservedRunningTime="2026-07-16 22:33:10.000856942 +0000 UTC m=+2644.837910742"
Jul 16 22:33:10.988657 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:10.988617    2555 generic.go:356] "Generic (PLEG): container finished" podID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerID="e066ea3e16f358f513df0463a5faba59986d8c79292abc6afafebd42bb0b9559" exitCode=6
Jul 16 22:33:10.989060 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:10.988706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerDied","Data":"e066ea3e16f358f513df0463a5faba59986d8c79292abc6afafebd42bb0b9559"}
Jul 16 22:33:10.989060 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:10.988909    2555 scope.go:117] "RemoveContainer" containerID="e066ea3e16f358f513df0463a5faba59986d8c79292abc6afafebd42bb0b9559"
Jul 16 22:33:11.995792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:11.995761    2555 generic.go:356] "Generic (PLEG): container finished" podID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerID="f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a" exitCode=6
Jul 16 22:33:11.996219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:11.995855    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerDied","Data":"f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a"}
Jul 16 22:33:11.996219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:11.995915    2555 scope.go:117] "RemoveContainer" containerID="e066ea3e16f358f513df0463a5faba59986d8c79292abc6afafebd42bb0b9559"
Jul 16 22:33:11.996219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:11.996031    2555 scope.go:117] "RemoveContainer" containerID="f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a"
Jul 16 22:33:11.996422 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:11.996224    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4_odh-ai-gateway-infra(0efbe6e6-fed7-4fe0-a40d-bfd20164aef3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3"
Jul 16 22:33:13.003301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:13.003274    2555 scope.go:117] "RemoveContainer" containerID="f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a"
Jul 16 22:33:13.003677 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:13.003504    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4_odh-ai-gateway-infra(0efbe6e6-fed7-4fe0-a40d-bfd20164aef3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3"
Jul 16 22:33:14.670451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:14.670418    2555 scope.go:117] "RemoveContainer" containerID="b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01"
Jul 16 22:33:16.023974 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:16.023935    2555 generic.go:356] "Generic (PLEG): container finished" podID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerID="27ceabddaf0b34fc7f1433862738ceb8f1104529633480f2a5e0f79b28459414" exitCode=6
Jul 16 22:33:16.024443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:16.024013    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" event={"ID":"fd752372-153d-41d0-a275-f8cbc18a5f41","Type":"ContainerDied","Data":"27ceabddaf0b34fc7f1433862738ceb8f1104529633480f2a5e0f79b28459414"}
Jul 16 22:33:16.024443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:16.024078    2555 scope.go:117] "RemoveContainer" containerID="b92b20a83a36158b95e6fadb1b2287dc2039056dd9e28a67c2dad624185b1d01"
Jul 16 22:33:16.024443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:16.024339    2555 scope.go:117] "RemoveContainer" containerID="27ceabddaf0b34fc7f1433862738ceb8f1104529633480f2a5e0f79b28459414"
Jul 16 22:33:16.024606 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:16.024576    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5_odh-ai-gateway-infra(fd752372-153d-41d0-a275-f8cbc18a5f41)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41"
Jul 16 22:33:17.056668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:17.056625    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:17.191695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:17.191670    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:17.223998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:17.223968    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca\") pod \"fd752372-153d-41d0-a275-f8cbc18a5f41\" (UID: \"fd752372-153d-41d0-a275-f8cbc18a5f41\") "
Jul 16 22:33:17.224299 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:17.224274    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fd752372-153d-41d0-a275-f8cbc18a5f41" (UID: "fd752372-153d-41d0-a275-f8cbc18a5f41"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:17.324785 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:17.324695    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fd752372-153d-41d0-a275-f8cbc18a5f41-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:33:18.037452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:18.037421    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"
Jul 16 22:33:18.037671 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:18.037475    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5" event={"ID":"fd752372-153d-41d0-a275-f8cbc18a5f41","Type":"ContainerDied","Data":"3c60013ba16b33286e441e382c25cf8b8f91956fb66e478f6371ca0a22cc4ea7"}
Jul 16 22:33:18.037671 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:18.037557    2555 scope.go:117] "RemoveContainer" containerID="27ceabddaf0b34fc7f1433862738ceb8f1104529633480f2a5e0f79b28459414"
Jul 16 22:33:18.062194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:18.062168    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:18.066398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:18.066366    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-nvbg5"]
Jul 16 22:33:19.676851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:19.676819    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" path="/var/lib/kubelet/pods/fd752372-153d-41d0-a275-f8cbc18a5f41/volumes"
Jul 16 22:33:25.673954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:25.673918    2555 scope.go:117] "RemoveContainer" containerID="f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a"
Jul 16 22:33:26.086059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:26.086027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerStarted","Data":"3112107f32fe8854438c071bcfb66b69b7c0e9e2fc613b2dd3a62d601eb18019"}
Jul 16 22:33:27.093288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:27.093253    2555 generic.go:356] "Generic (PLEG): container finished" podID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerID="3112107f32fe8854438c071bcfb66b69b7c0e9e2fc613b2dd3a62d601eb18019" exitCode=6
Jul 16 22:33:27.093783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:27.093325    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerDied","Data":"3112107f32fe8854438c071bcfb66b69b7c0e9e2fc613b2dd3a62d601eb18019"}
Jul 16 22:33:27.093783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:27.093404    2555 scope.go:117] "RemoveContainer" containerID="f44efaa616e4bcfb00a0d89268aee6ca5534902001ba35c8978490250384cd6a"
Jul 16 22:33:27.093783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:27.093769    2555 scope.go:117] "RemoveContainer" containerID="3112107f32fe8854438c071bcfb66b69b7c0e9e2fc613b2dd3a62d601eb18019"
Jul 16 22:33:27.094005 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:27.093988    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4_odh-ai-gateway-infra(0efbe6e6-fed7-4fe0-a40d-bfd20164aef3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3"
Jul 16 22:33:27.118138 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:27.118101    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:28.254564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:28.254536    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:28.321939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:28.321900    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca\") pod \"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3\" (UID: \"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3\") "
Jul 16 22:33:28.322267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:28.322243    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" (UID: "0efbe6e6-fed7-4fe0-a40d-bfd20164aef3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:28.423442 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:28.423357    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:33:29.106555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.106518    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4" event={"ID":"0efbe6e6-fed7-4fe0-a40d-bfd20164aef3","Type":"ContainerDied","Data":"640a1f09b7cbc1ed1ff1c5d7628f0cb2b8ac2ccba8ac0537b88a178bdad01821"}
Jul 16 22:33:29.106555 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.106553    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"
Jul 16 22:33:29.106799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.106593    2555 scope.go:117] "RemoveContainer" containerID="3112107f32fe8854438c071bcfb66b69b7c0e9e2fc613b2dd3a62d601eb18019"
Jul 16 22:33:29.148366 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.148330    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:29.154212 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.154180    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-gntt4"]
Jul 16 22:33:29.208824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.208791    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:29.209228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209205    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209229    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209242    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209251    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209264    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209272    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209287    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209299    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209315    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209323    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209336    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209344    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209434    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209448    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209457    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209483    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.209683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209490    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd752372-153d-41d0-a275-f8cbc18a5f41" containerName="revoke-keys"
Jul 16 22:33:29.209683 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.209499    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" containerName="revoke-keys"
Jul 16 22:33:29.213992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.213974    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:29.216948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.216931    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:33:29.217029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.216958    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:29.221075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.221047    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:29.332222 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.332176    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w\" (UID: \"c90943af-da27-416e-a46f-7177a5155813\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:29.432784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.432707    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w\" (UID: \"c90943af-da27-416e-a46f-7177a5155813\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:29.433324 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.433306    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w\" (UID: \"c90943af-da27-416e-a46f-7177a5155813\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:29.527201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.527173    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:29.670097 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:33:29.670070    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc90943af_da27_416e_a46f_7177a5155813.slice/crio-91c4a2a03b64727e4a1e4fe23323016e8e1a66dbc3a15d9ebc2046c36759d520 WatchSource:0}: Error finding container 91c4a2a03b64727e4a1e4fe23323016e8e1a66dbc3a15d9ebc2046c36759d520: Status 404 returned error can't find the container with id 91c4a2a03b64727e4a1e4fe23323016e8e1a66dbc3a15d9ebc2046c36759d520
Jul 16 22:33:29.682182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.682156    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0efbe6e6-fed7-4fe0-a40d-bfd20164aef3" path="/var/lib/kubelet/pods/0efbe6e6-fed7-4fe0-a40d-bfd20164aef3/volumes"
Jul 16 22:33:29.682442 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:29.682429    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:30.114392 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:30.114354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerStarted","Data":"ecb7cc3d64794470779b92cd040fb800bedba9ec3d2546b122b5404f0ef382de"}
Jul 16 22:33:30.114619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:30.114405    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerStarted","Data":"91c4a2a03b64727e4a1e4fe23323016e8e1a66dbc3a15d9ebc2046c36759d520"}
Jul 16 22:33:30.131234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:30.131167    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" podStartSLOduration=1.131149057 podStartE2EDuration="1.131149057s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:30.130911853 +0000 UTC m=+2664.967965654" watchObservedRunningTime="2026-07-16 22:33:30.131149057 +0000 UTC m=+2664.968202897"
Jul 16 22:33:31.121190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:31.121158    2555 generic.go:356] "Generic (PLEG): container finished" podID="c90943af-da27-416e-a46f-7177a5155813" containerID="ecb7cc3d64794470779b92cd040fb800bedba9ec3d2546b122b5404f0ef382de" exitCode=6
Jul 16 22:33:31.121652 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:31.121239    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerDied","Data":"ecb7cc3d64794470779b92cd040fb800bedba9ec3d2546b122b5404f0ef382de"}
Jul 16 22:33:31.121652 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:31.121418    2555 scope.go:117] "RemoveContainer" containerID="ecb7cc3d64794470779b92cd040fb800bedba9ec3d2546b122b5404f0ef382de"
Jul 16 22:33:32.128578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:32.128487    2555 generic.go:356] "Generic (PLEG): container finished" podID="c90943af-da27-416e-a46f-7177a5155813" containerID="42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca" exitCode=6
Jul 16 22:33:32.128989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:32.128623    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerDied","Data":"42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca"}
Jul 16 22:33:32.128989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:32.128675    2555 scope.go:117] "RemoveContainer" containerID="ecb7cc3d64794470779b92cd040fb800bedba9ec3d2546b122b5404f0ef382de"
Jul 16 22:33:32.128989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:32.128894    2555 scope.go:117] "RemoveContainer" containerID="42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca"
Jul 16 22:33:32.129171 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:32.129105    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w_odh-ai-gateway-infra(c90943af-da27-416e-a46f-7177a5155813)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" podUID="c90943af-da27-416e-a46f-7177a5155813"
Jul 16 22:33:33.135149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:33.135122    2555 scope.go:117] "RemoveContainer" containerID="42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca"
Jul 16 22:33:33.135582 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:33.135335    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w_odh-ai-gateway-infra(c90943af-da27-416e-a46f-7177a5155813)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" podUID="c90943af-da27-416e-a46f-7177a5155813"
Jul 16 22:33:45.673493 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:45.673448    2555 scope.go:117] "RemoveContainer" containerID="42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca"
Jul 16 22:33:46.212434 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:46.212400    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerStarted","Data":"92af828f516c1337cf6bd976bf168a325cffa34d6fd122e39ff4f66ec794bfe2"}
Jul 16 22:33:47.219782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:47.219737    2555 generic.go:356] "Generic (PLEG): container finished" podID="c90943af-da27-416e-a46f-7177a5155813" containerID="92af828f516c1337cf6bd976bf168a325cffa34d6fd122e39ff4f66ec794bfe2" exitCode=6
Jul 16 22:33:47.220272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:47.219809    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerDied","Data":"92af828f516c1337cf6bd976bf168a325cffa34d6fd122e39ff4f66ec794bfe2"}
Jul 16 22:33:47.220272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:47.219903    2555 scope.go:117] "RemoveContainer" containerID="42b4af6a61081bee2ec0ba38a527e7bafb74fda4427caf269696e5a571a612ca"
Jul 16 22:33:47.220272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:47.220229    2555 scope.go:117] "RemoveContainer" containerID="92af828f516c1337cf6bd976bf168a325cffa34d6fd122e39ff4f66ec794bfe2"
Jul 16 22:33:47.220522 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:47.220500    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w_odh-ai-gateway-infra(c90943af-da27-416e-a46f-7177a5155813)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" podUID="c90943af-da27-416e-a46f-7177a5155813"
Jul 16 22:33:47.269721 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:47.269685    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:48.363443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:48.363416    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:48.498624 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:48.498538    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca\") pod \"c90943af-da27-416e-a46f-7177a5155813\" (UID: \"c90943af-da27-416e-a46f-7177a5155813\") "
Jul 16 22:33:48.498953 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:48.498931    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c90943af-da27-416e-a46f-7177a5155813" (UID: "c90943af-da27-416e-a46f-7177a5155813"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:33:48.599833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:48.599797    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c90943af-da27-416e-a46f-7177a5155813-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:33:49.234990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.234953    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"
Jul 16 22:33:49.234990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.234972    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w" event={"ID":"c90943af-da27-416e-a46f-7177a5155813","Type":"ContainerDied","Data":"91c4a2a03b64727e4a1e4fe23323016e8e1a66dbc3a15d9ebc2046c36759d520"}
Jul 16 22:33:49.235228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.235038    2555 scope.go:117] "RemoveContainer" containerID="92af828f516c1337cf6bd976bf168a325cffa34d6fd122e39ff4f66ec794bfe2"
Jul 16 22:33:49.257850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.257821    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:49.261564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.261538    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-fqg8w"]
Jul 16 22:33:49.676269 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:49.676226    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c90943af-da27-416e-a46f-7177a5155813" path="/var/lib/kubelet/pods/c90943af-da27-416e-a46f-7177a5155813/volumes"
Jul 16 22:33:50.399779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.399744    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:33:50.400147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400130    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400150    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400174    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400182    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400192    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400267 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400210    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400332    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.400532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.400343    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:33:50.404692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.404669    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:33:50.407215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.407195    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:33:50.407310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.407266    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:33:50.411723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.411700    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:33:50.515203 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.515164    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7\" (UID: \"dc52d774-e33a-429c-a854-3c048e17c161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:33:50.616026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.615985    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7\" (UID: \"dc52d774-e33a-429c-a854-3c048e17c161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:33:50.616647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.616629    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7\" (UID: \"dc52d774-e33a-429c-a854-3c048e17c161\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:33:50.716659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.716571    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:33:50.850812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:50.850783    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:33:50.852156 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:33:50.852128    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc52d774_e33a_429c_a854_3c048e17c161.slice/crio-2b111aba9651e1b3f4dd9691c48ecca4a70be7b6038eb28e9acb9a4d9e9dc47b WatchSource:0}: Error finding container 2b111aba9651e1b3f4dd9691c48ecca4a70be7b6038eb28e9acb9a4d9e9dc47b: Status 404 returned error can't find the container with id 2b111aba9651e1b3f4dd9691c48ecca4a70be7b6038eb28e9acb9a4d9e9dc47b
Jul 16 22:33:51.248099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:51.248064    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerStarted","Data":"1568e0af3720e5a8dbd4c01cd74d92747961adad110c3279eedadee5b4369bb9"}
Jul 16 22:33:51.248321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:51.248125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerStarted","Data":"2b111aba9651e1b3f4dd9691c48ecca4a70be7b6038eb28e9acb9a4d9e9dc47b"}
Jul 16 22:33:51.265647 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:51.265535    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" podStartSLOduration=1.2654574140000001 podStartE2EDuration="1.265457414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:33:51.264191697 +0000 UTC m=+2686.101245498" watchObservedRunningTime="2026-07-16 22:33:51.265457414 +0000 UTC m=+2686.102511248"
Jul 16 22:33:52.255095 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:52.255053    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc52d774-e33a-429c-a854-3c048e17c161" containerID="1568e0af3720e5a8dbd4c01cd74d92747961adad110c3279eedadee5b4369bb9" exitCode=6
Jul 16 22:33:52.255530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:52.255138    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerDied","Data":"1568e0af3720e5a8dbd4c01cd74d92747961adad110c3279eedadee5b4369bb9"}
Jul 16 22:33:52.255530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:52.255302    2555 scope.go:117] "RemoveContainer" containerID="1568e0af3720e5a8dbd4c01cd74d92747961adad110c3279eedadee5b4369bb9"
Jul 16 22:33:53.263227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:53.263142    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc52d774-e33a-429c-a854-3c048e17c161" containerID="7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794" exitCode=6
Jul 16 22:33:53.263672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:53.263240    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerDied","Data":"7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794"}
Jul 16 22:33:53.263672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:53.263315    2555 scope.go:117] "RemoveContainer" containerID="1568e0af3720e5a8dbd4c01cd74d92747961adad110c3279eedadee5b4369bb9"
Jul 16 22:33:53.263672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:53.263486    2555 scope.go:117] "RemoveContainer" containerID="7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794"
Jul 16 22:33:53.263849 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:53.263710    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7_odh-ai-gateway-infra(dc52d774-e33a-429c-a854-3c048e17c161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" podUID="dc52d774-e33a-429c-a854-3c048e17c161"
Jul 16 22:33:54.271043 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:54.271015    2555 scope.go:117] "RemoveContainer" containerID="7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794"
Jul 16 22:33:54.271419 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:33:54.271195    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7_odh-ai-gateway-infra(dc52d774-e33a-429c-a854-3c048e17c161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" podUID="dc52d774-e33a-429c-a854-3c048e17c161"
Jul 16 22:33:59.189730 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:33:59.189644    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:34:06.671047 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:06.671009    2555 scope.go:117] "RemoveContainer" containerID="7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794"
Jul 16 22:34:07.341984 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:07.341953    2555 generic.go:356] "Generic (PLEG): container finished" podID="dc52d774-e33a-429c-a854-3c048e17c161" containerID="8d4a07f7cbefad65080ae5cd4a169c6c50b9fe36a9a4f2769aebf762f8952300" exitCode=6
Jul 16 22:34:07.342106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:07.342029    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerDied","Data":"8d4a07f7cbefad65080ae5cd4a169c6c50b9fe36a9a4f2769aebf762f8952300"}
Jul 16 22:34:07.342106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:07.342086    2555 scope.go:117] "RemoveContainer" containerID="7609b551b566e306603a9cd7c46d7a7ce747b8e7a358c94158fbb67cdeac9794"
Jul 16 22:34:07.342414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:07.342395    2555 scope.go:117] "RemoveContainer" containerID="8d4a07f7cbefad65080ae5cd4a169c6c50b9fe36a9a4f2769aebf762f8952300"
Jul 16 22:34:07.342656 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:07.342631    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7_odh-ai-gateway-infra(dc52d774-e33a-429c-a854-3c048e17c161)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" podUID="dc52d774-e33a-429c-a854-3c048e17c161"
Jul 16 22:34:08.373385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:08.368167    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:34:08.512388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:08.512363    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:34:08.564332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:08.564305    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca\") pod \"dc52d774-e33a-429c-a854-3c048e17c161\" (UID: \"dc52d774-e33a-429c-a854-3c048e17c161\") "
Jul 16 22:34:08.564708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:08.564685    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "dc52d774-e33a-429c-a854-3c048e17c161" (UID: "dc52d774-e33a-429c-a854-3c048e17c161"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:08.665301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:08.665229    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/dc52d774-e33a-429c-a854-3c048e17c161-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:34:09.155992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.155955    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:09.156340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156326    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156342    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156351    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156357    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156422    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156429    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156436    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:09.156523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.156441    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c90943af-da27-416e-a46f-7177a5155813" containerName="revoke-keys"
Jul 16 22:34:09.159570 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.159553    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:09.166505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.166479    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:09.168086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.168058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg\" (UID: \"9e01ec18-006c-41be-a0c8-cc2c7683790b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:09.268793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.268754    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg\" (UID: \"9e01ec18-006c-41be-a0c8-cc2c7683790b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:09.269420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.269399    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg\" (UID: \"9e01ec18-006c-41be-a0c8-cc2c7683790b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:09.355673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.355638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7" event={"ID":"dc52d774-e33a-429c-a854-3c048e17c161","Type":"ContainerDied","Data":"2b111aba9651e1b3f4dd9691c48ecca4a70be7b6038eb28e9acb9a4d9e9dc47b"}
Jul 16 22:34:09.355673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.355674    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"
Jul 16 22:34:09.355876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.355698    2555 scope.go:117] "RemoveContainer" containerID="8d4a07f7cbefad65080ae5cd4a169c6c50b9fe36a9a4f2769aebf762f8952300"
Jul 16 22:34:09.382214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.382183    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:34:09.387174 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.387151    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-rhgh7"]
Jul 16 22:34:09.470745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.470677    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:09.607010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.606754    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:09.611027 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:34:09.610998    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e01ec18_006c_41be_a0c8_cc2c7683790b.slice/crio-e12fcfc093a120285112afff6da00ae5413cb2d6195b5c09b3ecd4a5362aac61 WatchSource:0}: Error finding container e12fcfc093a120285112afff6da00ae5413cb2d6195b5c09b3ecd4a5362aac61: Status 404 returned error can't find the container with id e12fcfc093a120285112afff6da00ae5413cb2d6195b5c09b3ecd4a5362aac61
Jul 16 22:34:09.676264 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:09.676241    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dc52d774-e33a-429c-a854-3c048e17c161" path="/var/lib/kubelet/pods/dc52d774-e33a-429c-a854-3c048e17c161/volumes"
Jul 16 22:34:10.362928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:10.362893    2555 generic.go:356] "Generic (PLEG): container finished" podID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerID="a1c97f6add5ad834273c9409a82282513ce88a4ab2dc092e1c1eacdc478a7f27" exitCode=6
Jul 16 22:34:10.363126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:10.362988    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" event={"ID":"9e01ec18-006c-41be-a0c8-cc2c7683790b","Type":"ContainerDied","Data":"a1c97f6add5ad834273c9409a82282513ce88a4ab2dc092e1c1eacdc478a7f27"}
Jul 16 22:34:10.363126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:10.363041    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" event={"ID":"9e01ec18-006c-41be-a0c8-cc2c7683790b","Type":"ContainerStarted","Data":"e12fcfc093a120285112afff6da00ae5413cb2d6195b5c09b3ecd4a5362aac61"}
Jul 16 22:34:10.363246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:10.363165    2555 scope.go:117] "RemoveContainer" containerID="a1c97f6add5ad834273c9409a82282513ce88a4ab2dc092e1c1eacdc478a7f27"
Jul 16 22:34:11.371014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:11.370978    2555 generic.go:356] "Generic (PLEG): container finished" podID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerID="3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff" exitCode=6
Jul 16 22:34:11.371481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:11.371043    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" event={"ID":"9e01ec18-006c-41be-a0c8-cc2c7683790b","Type":"ContainerDied","Data":"3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff"}
Jul 16 22:34:11.371481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:11.371099    2555 scope.go:117] "RemoveContainer" containerID="a1c97f6add5ad834273c9409a82282513ce88a4ab2dc092e1c1eacdc478a7f27"
Jul 16 22:34:11.371481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:11.371275    2555 scope.go:117] "RemoveContainer" containerID="3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff"
Jul 16 22:34:11.371607 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:11.371497    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg_odh-ai-gateway-infra(9e01ec18-006c-41be-a0c8-cc2c7683790b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b"
Jul 16 22:34:12.379935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:12.379911    2555 scope.go:117] "RemoveContainer" containerID="3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff"
Jul 16 22:34:12.380324 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:12.380096    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg_odh-ai-gateway-infra(9e01ec18-006c-41be-a0c8-cc2c7683790b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b"
Jul 16 22:34:14.083819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:14.083787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:34:26.670871 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:26.670839    2555 scope.go:117] "RemoveContainer" containerID="3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff"
Jul 16 22:34:27.459188 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:27.459151    2555 generic.go:356] "Generic (PLEG): container finished" podID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerID="d9fa548793edc1f1d326938ac6992a29ba337ef73f09aa386e141e0a97875eb7" exitCode=6
Jul 16 22:34:27.459352 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:27.459212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" event={"ID":"9e01ec18-006c-41be-a0c8-cc2c7683790b","Type":"ContainerDied","Data":"d9fa548793edc1f1d326938ac6992a29ba337ef73f09aa386e141e0a97875eb7"}
Jul 16 22:34:27.459352 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:27.459267    2555 scope.go:117] "RemoveContainer" containerID="3c9bf2c34095acbe0bbb50db98150a95d0281660ea22bc76c270c625869442ff"
Jul 16 22:34:27.459575 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:27.459554    2555 scope.go:117] "RemoveContainer" containerID="d9fa548793edc1f1d326938ac6992a29ba337ef73f09aa386e141e0a97875eb7"
Jul 16 22:34:27.459822 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:27.459769    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg_odh-ai-gateway-infra(9e01ec18-006c-41be-a0c8-cc2c7683790b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b"
Jul 16 22:34:28.506048 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:28.506014    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:28.645185 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:28.645159    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:28.735180 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:28.735145    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca\") pod \"9e01ec18-006c-41be-a0c8-cc2c7683790b\" (UID: \"9e01ec18-006c-41be-a0c8-cc2c7683790b\") "
Jul 16 22:34:28.735569 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:28.735540    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e01ec18-006c-41be-a0c8-cc2c7683790b" (UID: "9e01ec18-006c-41be-a0c8-cc2c7683790b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:28.835772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:28.835741    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e01ec18-006c-41be-a0c8-cc2c7683790b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:34:29.253572 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253488    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:29.253837 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253825    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.253892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253839    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.253892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253849    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.253892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.253892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253874    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:29.253892 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253880    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="dc52d774-e33a-429c-a854-3c048e17c161" containerName="revoke-keys"
Jul 16 22:34:29.254056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253929    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.254056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253936    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.254056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.253942    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:29.257032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.257016    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:29.264986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.264794    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:29.339791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.339742    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z444k\" (UID: \"725e7891-821a-49d9-9267-1ccd37c70cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:29.440533 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.440502    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z444k\" (UID: \"725e7891-821a-49d9-9267-1ccd37c70cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:29.441114 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.441096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-z444k\" (UID: \"725e7891-821a-49d9-9267-1ccd37c70cdb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:29.472349 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.472327    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"
Jul 16 22:34:29.472349 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.472325    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg" event={"ID":"9e01ec18-006c-41be-a0c8-cc2c7683790b","Type":"ContainerDied","Data":"e12fcfc093a120285112afff6da00ae5413cb2d6195b5c09b3ecd4a5362aac61"}
Jul 16 22:34:29.472621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.472395    2555 scope.go:117] "RemoveContainer" containerID="d9fa548793edc1f1d326938ac6992a29ba337ef73f09aa386e141e0a97875eb7"
Jul 16 22:34:29.496020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.495990    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:29.499094 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.499071    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fd5mg"]
Jul 16 22:34:29.570579 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.570549    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:29.679718 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.679684    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" path="/var/lib/kubelet/pods/9e01ec18-006c-41be-a0c8-cc2c7683790b/volumes"
Jul 16 22:34:29.708386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:29.708343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:29.708838 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:34:29.708812    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod725e7891_821a_49d9_9267_1ccd37c70cdb.slice/crio-c82e6dfff19f005a05f72c30d362cddee1f77f4ecf12d949adf58b1325074209 WatchSource:0}: Error finding container c82e6dfff19f005a05f72c30d362cddee1f77f4ecf12d949adf58b1325074209: Status 404 returned error can't find the container with id c82e6dfff19f005a05f72c30d362cddee1f77f4ecf12d949adf58b1325074209
Jul 16 22:34:30.479452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:30.479413    2555 generic.go:356] "Generic (PLEG): container finished" podID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerID="5b25e6e2e0871b9ee26e764cdf9c86acc1e67455bcf2917e1e517ad50e232688" exitCode=6
Jul 16 22:34:30.479707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:30.479507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" event={"ID":"725e7891-821a-49d9-9267-1ccd37c70cdb","Type":"ContainerDied","Data":"5b25e6e2e0871b9ee26e764cdf9c86acc1e67455bcf2917e1e517ad50e232688"}
Jul 16 22:34:30.479707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:30.479571    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" event={"ID":"725e7891-821a-49d9-9267-1ccd37c70cdb","Type":"ContainerStarted","Data":"c82e6dfff19f005a05f72c30d362cddee1f77f4ecf12d949adf58b1325074209"}
Jul 16 22:34:30.479707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:30.479701    2555 scope.go:117] "RemoveContainer" containerID="5b25e6e2e0871b9ee26e764cdf9c86acc1e67455bcf2917e1e517ad50e232688"
Jul 16 22:34:31.488945 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:31.488855    2555 generic.go:356] "Generic (PLEG): container finished" podID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerID="d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f" exitCode=6
Jul 16 22:34:31.488945 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:31.488931    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" event={"ID":"725e7891-821a-49d9-9267-1ccd37c70cdb","Type":"ContainerDied","Data":"d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f"}
Jul 16 22:34:31.489449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:31.488972    2555 scope.go:117] "RemoveContainer" containerID="5b25e6e2e0871b9ee26e764cdf9c86acc1e67455bcf2917e1e517ad50e232688"
Jul 16 22:34:31.489449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:31.489175    2555 scope.go:117] "RemoveContainer" containerID="d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f"
Jul 16 22:34:31.489449 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:31.489402    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z444k_odh-ai-gateway-infra(725e7891-821a-49d9-9267-1ccd37c70cdb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb"
Jul 16 22:34:32.495525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:32.495498    2555 scope.go:117] "RemoveContainer" containerID="d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f"
Jul 16 22:34:32.495924 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:32.495720    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z444k_odh-ai-gateway-infra(725e7891-821a-49d9-9267-1ccd37c70cdb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb"
Jul 16 22:34:40.478357 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.478320    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:40.480760 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.478710    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:40.480760 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.478722    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e01ec18-006c-41be-a0c8-cc2c7683790b" containerName="revoke-keys"
Jul 16 22:34:40.481620 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.481604    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:40.522380 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.522357    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:40.536234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.536202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6\" (UID: \"7ffa2b01-528b-4351-86a0-23e6fda38933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:40.637793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.637755    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6\" (UID: \"7ffa2b01-528b-4351-86a0-23e6fda38933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:40.638434 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.638410    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6\" (UID: \"7ffa2b01-528b-4351-86a0-23e6fda38933\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:40.791651 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.791606    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:40.935315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:40.935281    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:40.936543 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:34:40.936516    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ffa2b01_528b_4351_86a0_23e6fda38933.slice/crio-9ffdfa4a48fc8f0ae4bbfd05b56c6465e445034958adb3a3b31aac242082fb3e WatchSource:0}: Error finding container 9ffdfa4a48fc8f0ae4bbfd05b56c6465e445034958adb3a3b31aac242082fb3e: Status 404 returned error can't find the container with id 9ffdfa4a48fc8f0ae4bbfd05b56c6465e445034958adb3a3b31aac242082fb3e
Jul 16 22:34:41.551885 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:41.551856    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerID="4f8946f3e3b12845c6a6bfac2d3da3d92f305fe18bfcfc6af8682a550bb282d0" exitCode=6
Jul 16 22:34:41.552305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:41.551942    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" event={"ID":"7ffa2b01-528b-4351-86a0-23e6fda38933","Type":"ContainerDied","Data":"4f8946f3e3b12845c6a6bfac2d3da3d92f305fe18bfcfc6af8682a550bb282d0"}
Jul 16 22:34:41.552305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:41.552000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" event={"ID":"7ffa2b01-528b-4351-86a0-23e6fda38933","Type":"ContainerStarted","Data":"9ffdfa4a48fc8f0ae4bbfd05b56c6465e445034958adb3a3b31aac242082fb3e"}
Jul 16 22:34:41.552305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:41.552089    2555 scope.go:117] "RemoveContainer" containerID="4f8946f3e3b12845c6a6bfac2d3da3d92f305fe18bfcfc6af8682a550bb282d0"
Jul 16 22:34:42.559428 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:42.559400    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerID="859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45" exitCode=6
Jul 16 22:34:42.559821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:42.559494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" event={"ID":"7ffa2b01-528b-4351-86a0-23e6fda38933","Type":"ContainerDied","Data":"859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45"}
Jul 16 22:34:42.559821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:42.559559    2555 scope.go:117] "RemoveContainer" containerID="4f8946f3e3b12845c6a6bfac2d3da3d92f305fe18bfcfc6af8682a550bb282d0"
Jul 16 22:34:42.559821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:42.559786    2555 scope.go:117] "RemoveContainer" containerID="859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45"
Jul 16 22:34:42.559997 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:42.559978    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6_odh-ai-gateway-infra(7ffa2b01-528b-4351-86a0-23e6fda38933)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933"
Jul 16 22:34:43.568578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:43.568551    2555 scope.go:117] "RemoveContainer" containerID="859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45"
Jul 16 22:34:43.568962 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:43.568745    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6_odh-ai-gateway-infra(7ffa2b01-528b-4351-86a0-23e6fda38933)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933"
Jul 16 22:34:43.670751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:43.670720    2555 scope.go:117] "RemoveContainer" containerID="d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f"
Jul 16 22:34:44.576164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:44.576077    2555 generic.go:356] "Generic (PLEG): container finished" podID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerID="8629b06b38df02df534ca08ab0046b0e0221fe28eef599cb14ac2d03567490d3" exitCode=6
Jul 16 22:34:44.576164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:44.576137    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" event={"ID":"725e7891-821a-49d9-9267-1ccd37c70cdb","Type":"ContainerDied","Data":"8629b06b38df02df534ca08ab0046b0e0221fe28eef599cb14ac2d03567490d3"}
Jul 16 22:34:44.576680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:44.576183    2555 scope.go:117] "RemoveContainer" containerID="d83fdbd0f664eb72922d288a7feb716688c8d30d69522f47867feaebe80a7c5f"
Jul 16 22:34:44.576680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:44.576506    2555 scope.go:117] "RemoveContainer" containerID="8629b06b38df02df534ca08ab0046b0e0221fe28eef599cb14ac2d03567490d3"
Jul 16 22:34:44.576823 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:44.576720    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-z444k_odh-ai-gateway-infra(725e7891-821a-49d9-9267-1ccd37c70cdb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb"
Jul 16 22:34:45.603000 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:45.602966    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:45.744288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:45.744262    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:45.887388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:45.887293    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca\") pod \"725e7891-821a-49d9-9267-1ccd37c70cdb\" (UID: \"725e7891-821a-49d9-9267-1ccd37c70cdb\") "
Jul 16 22:34:45.887720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:45.887693    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "725e7891-821a-49d9-9267-1ccd37c70cdb" (UID: "725e7891-821a-49d9-9267-1ccd37c70cdb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:45.988006 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:45.987969    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/725e7891-821a-49d9-9267-1ccd37c70cdb-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:34:46.589756 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:46.589718    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k" event={"ID":"725e7891-821a-49d9-9267-1ccd37c70cdb","Type":"ContainerDied","Data":"c82e6dfff19f005a05f72c30d362cddee1f77f4ecf12d949adf58b1325074209"}
Jul 16 22:34:46.589970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:46.589760    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"
Jul 16 22:34:46.589970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:46.589788    2555 scope.go:117] "RemoveContainer" containerID="8629b06b38df02df534ca08ab0046b0e0221fe28eef599cb14ac2d03567490d3"
Jul 16 22:34:46.612422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:46.612398    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:46.619609 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:46.619581    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-z444k"]
Jul 16 22:34:47.677322 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:47.677292    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" path="/var/lib/kubelet/pods/725e7891-821a-49d9-9267-1ccd37c70cdb/volumes"
Jul 16 22:34:51.063485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063424    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063821    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063840    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063856    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063861    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063868    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.063879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063873    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.064118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063931    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.064118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.063938    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:34:51.068208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.068188    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:34:51.076534 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.076504    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:34:51.237938 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.237897    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ckb4p\" (UID: \"49b2c5e5-df18-4768-b634-ed49d2b9c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:34:51.339072 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.338977    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ckb4p\" (UID: \"49b2c5e5-df18-4768-b634-ed49d2b9c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:34:51.339619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.339601    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ckb4p\" (UID: \"49b2c5e5-df18-4768-b634-ed49d2b9c65e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:34:51.380012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.379988    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:34:51.535316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.535288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:34:51.537644 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:34:51.537614    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b2c5e5_df18_4768_b634_ed49d2b9c65e.slice/crio-eba0356c78fce15bd00469b54aa6fd01c6b2fd71ed661c5f43d94e963e7a399d WatchSource:0}: Error finding container eba0356c78fce15bd00469b54aa6fd01c6b2fd71ed661c5f43d94e963e7a399d: Status 404 returned error can't find the container with id eba0356c78fce15bd00469b54aa6fd01c6b2fd71ed661c5f43d94e963e7a399d
Jul 16 22:34:51.619917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:51.619831    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" event={"ID":"49b2c5e5-df18-4768-b634-ed49d2b9c65e","Type":"ContainerStarted","Data":"eba0356c78fce15bd00469b54aa6fd01c6b2fd71ed661c5f43d94e963e7a399d"}
Jul 16 22:34:52.626607 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:52.626571    2555 generic.go:356] "Generic (PLEG): container finished" podID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerID="d6ee006a8f75dbcac45973d909c9cc0a82a6b8670e61744ad89f630399d3186c" exitCode=6
Jul 16 22:34:52.627015 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:52.626645    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" event={"ID":"49b2c5e5-df18-4768-b634-ed49d2b9c65e","Type":"ContainerDied","Data":"d6ee006a8f75dbcac45973d909c9cc0a82a6b8670e61744ad89f630399d3186c"}
Jul 16 22:34:52.627015 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:52.626822    2555 scope.go:117] "RemoveContainer" containerID="d6ee006a8f75dbcac45973d909c9cc0a82a6b8670e61744ad89f630399d3186c"
Jul 16 22:34:53.236830 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:53.236793    2555 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b2c5e5_df18_4768_b634_ed49d2b9c65e.slice/crio-48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49b2c5e5_df18_4768_b634_ed49d2b9c65e.slice/crio-conmon-48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:34:53.634053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:53.634019    2555 generic.go:356] "Generic (PLEG): container finished" podID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerID="48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc" exitCode=6
Jul 16 22:34:53.634485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:53.634104    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" event={"ID":"49b2c5e5-df18-4768-b634-ed49d2b9c65e","Type":"ContainerDied","Data":"48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc"}
Jul 16 22:34:53.634485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:53.634162    2555 scope.go:117] "RemoveContainer" containerID="d6ee006a8f75dbcac45973d909c9cc0a82a6b8670e61744ad89f630399d3186c"
Jul 16 22:34:53.634485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:53.634312    2555 scope.go:117] "RemoveContainer" containerID="48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc"
Jul 16 22:34:53.634662 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:53.634564    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ckb4p_odh-ai-gateway-infra(49b2c5e5-df18-4768-b634-ed49d2b9c65e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e"
Jul 16 22:34:54.641142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:54.641114    2555 scope.go:117] "RemoveContainer" containerID="48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc"
Jul 16 22:34:54.641565 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:54.641321    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ckb4p_odh-ai-gateway-infra(49b2c5e5-df18-4768-b634-ed49d2b9c65e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e"
Jul 16 22:34:54.671294 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:54.670899    2555 scope.go:117] "RemoveContainer" containerID="859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45"
Jul 16 22:34:55.649906 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:55.649821    2555 generic.go:356] "Generic (PLEG): container finished" podID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerID="941bfd34ebf8ccae8f269ed7cf11678087a583f1f1f0c81a5613ad99695b3258" exitCode=6
Jul 16 22:34:55.649906 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:55.649897    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" event={"ID":"7ffa2b01-528b-4351-86a0-23e6fda38933","Type":"ContainerDied","Data":"941bfd34ebf8ccae8f269ed7cf11678087a583f1f1f0c81a5613ad99695b3258"}
Jul 16 22:34:55.650399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:55.649958    2555 scope.go:117] "RemoveContainer" containerID="859f27101835b37202f605e913988c8f94fbffc713c29a54383449e482946d45"
Jul 16 22:34:55.650399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:55.650261    2555 scope.go:117] "RemoveContainer" containerID="941bfd34ebf8ccae8f269ed7cf11678087a583f1f1f0c81a5613ad99695b3258"
Jul 16 22:34:55.650508 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:34:55.650450    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6_odh-ai-gateway-infra(7ffa2b01-528b-4351-86a0-23e6fda38933)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933"
Jul 16 22:34:56.688323 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:56.688289    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:56.826246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:56.826220    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:56.988590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:56.988494    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca\") pod \"7ffa2b01-528b-4351-86a0-23e6fda38933\" (UID: \"7ffa2b01-528b-4351-86a0-23e6fda38933\") "
Jul 16 22:34:56.988848 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:56.988825    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7ffa2b01-528b-4351-86a0-23e6fda38933" (UID: "7ffa2b01-528b-4351-86a0-23e6fda38933"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:34:57.090030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.089998    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ffa2b01-528b-4351-86a0-23e6fda38933-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:34:57.664407 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.664380    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"
Jul 16 22:34:57.664407 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.664402    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6" event={"ID":"7ffa2b01-528b-4351-86a0-23e6fda38933","Type":"ContainerDied","Data":"9ffdfa4a48fc8f0ae4bbfd05b56c6465e445034958adb3a3b31aac242082fb3e"}
Jul 16 22:34:57.664661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.664458    2555 scope.go:117] "RemoveContainer" containerID="941bfd34ebf8ccae8f269ed7cf11678087a583f1f1f0c81a5613ad99695b3258"
Jul 16 22:34:57.688122 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.688098    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:57.691916 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:57.691893    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-mvmf6"]
Jul 16 22:34:59.676538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:34:59.676509    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" path="/var/lib/kubelet/pods/7ffa2b01-528b-4351-86a0-23e6fda38933/volumes"
Jul 16 22:35:02.390249 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:02.390216    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:35:09.214597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.214566    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:09.214985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.214952    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.214985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.214965    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.214985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.214980    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.214985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.214986    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215001    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215006    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215068    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="725e7891-821a-49d9-9267-1ccd37c70cdb" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215080    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215086    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.215111 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.215092    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ffa2b01-528b-4351-86a0-23e6fda38933" containerName="revoke-keys"
Jul 16 22:35:09.219615 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.219597    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:09.228919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.228894    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:09.299005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.298969    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq\" (UID: \"845c2959-d054-47a6-8b05-996e01479535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:09.399667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.399631    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq\" (UID: \"845c2959-d054-47a6-8b05-996e01479535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:09.400316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.400296    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq\" (UID: \"845c2959-d054-47a6-8b05-996e01479535\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:09.530164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.530133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:09.676402 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:35:09.676373    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845c2959_d054_47a6_8b05_996e01479535.slice/crio-7d77984f96884f9a9c5c33ec04c50728bf80334ed1eb932a58a4f0cfa7c3b462 WatchSource:0}: Error finding container 7d77984f96884f9a9c5c33ec04c50728bf80334ed1eb932a58a4f0cfa7c3b462: Status 404 returned error can't find the container with id 7d77984f96884f9a9c5c33ec04c50728bf80334ed1eb932a58a4f0cfa7c3b462
Jul 16 22:35:09.678499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.678478    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:09.679035 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.678936    2555 scope.go:117] "RemoveContainer" containerID="48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc"
Jul 16 22:35:09.730923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:09.730895    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" event={"ID":"845c2959-d054-47a6-8b05-996e01479535","Type":"ContainerStarted","Data":"7d77984f96884f9a9c5c33ec04c50728bf80334ed1eb932a58a4f0cfa7c3b462"}
Jul 16 22:35:10.738061 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.737976    2555 generic.go:356] "Generic (PLEG): container finished" podID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerID="b7c081d23915cba56073aac193b45dcdfbfdf5342d92cd32d751dc2b5800e415" exitCode=6
Jul 16 22:35:10.738061 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.738044    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" event={"ID":"49b2c5e5-df18-4768-b634-ed49d2b9c65e","Type":"ContainerDied","Data":"b7c081d23915cba56073aac193b45dcdfbfdf5342d92cd32d751dc2b5800e415"}
Jul 16 22:35:10.738593 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.738120    2555 scope.go:117] "RemoveContainer" containerID="48dd3c14ec5fdef56ba19724c38a79fc85e9e2665ee9e897f6bada9cd31330cc"
Jul 16 22:35:10.738593 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.738426    2555 scope.go:117] "RemoveContainer" containerID="b7c081d23915cba56073aac193b45dcdfbfdf5342d92cd32d751dc2b5800e415"
Jul 16 22:35:10.738701 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:10.738678    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ckb4p_odh-ai-gateway-infra(49b2c5e5-df18-4768-b634-ed49d2b9c65e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e"
Jul 16 22:35:10.739726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.739704    2555 generic.go:356] "Generic (PLEG): container finished" podID="845c2959-d054-47a6-8b05-996e01479535" containerID="77786afd1ee3cc685efd4b8f9cd37b708774369f58b1e6f809b2dc243c5ca789" exitCode=6
Jul 16 22:35:10.739824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.739736    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" event={"ID":"845c2959-d054-47a6-8b05-996e01479535","Type":"ContainerDied","Data":"77786afd1ee3cc685efd4b8f9cd37b708774369f58b1e6f809b2dc243c5ca789"}
Jul 16 22:35:10.739910 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:10.739897    2555 scope.go:117] "RemoveContainer" containerID="77786afd1ee3cc685efd4b8f9cd37b708774369f58b1e6f809b2dc243c5ca789"
Jul 16 22:35:11.747058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.746969    2555 generic.go:356] "Generic (PLEG): container finished" podID="845c2959-d054-47a6-8b05-996e01479535" containerID="d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99" exitCode=6
Jul 16 22:35:11.747578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.747063    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" event={"ID":"845c2959-d054-47a6-8b05-996e01479535","Type":"ContainerDied","Data":"d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99"}
Jul 16 22:35:11.747578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.747124    2555 scope.go:117] "RemoveContainer" containerID="77786afd1ee3cc685efd4b8f9cd37b708774369f58b1e6f809b2dc243c5ca789"
Jul 16 22:35:11.747578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.747365    2555 scope.go:117] "RemoveContainer" containerID="d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99"
Jul 16 22:35:11.747747 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:11.747625    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq_odh-ai-gateway-infra(845c2959-d054-47a6-8b05-996e01479535)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" podUID="845c2959-d054-47a6-8b05-996e01479535"
Jul 16 22:35:11.767151 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.767124    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:35:11.902967 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.902940    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:35:11.920442 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.920411    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca\") pod \"49b2c5e5-df18-4768-b634-ed49d2b9c65e\" (UID: \"49b2c5e5-df18-4768-b634-ed49d2b9c65e\") "
Jul 16 22:35:11.920808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:11.920780    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "49b2c5e5-df18-4768-b634-ed49d2b9c65e" (UID: "49b2c5e5-df18-4768-b634-ed49d2b9c65e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:12.021716 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.021623    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/49b2c5e5-df18-4768-b634-ed49d2b9c65e-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:35:12.755143 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.755113    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"
Jul 16 22:35:12.755668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.755161    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p" event={"ID":"49b2c5e5-df18-4768-b634-ed49d2b9c65e","Type":"ContainerDied","Data":"eba0356c78fce15bd00469b54aa6fd01c6b2fd71ed661c5f43d94e963e7a399d"}
Jul 16 22:35:12.755668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.755230    2555 scope.go:117] "RemoveContainer" containerID="b7c081d23915cba56073aac193b45dcdfbfdf5342d92cd32d751dc2b5800e415"
Jul 16 22:35:12.757062 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.757042    2555 scope.go:117] "RemoveContainer" containerID="d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99"
Jul 16 22:35:12.757279 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:12.757253    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq_odh-ai-gateway-infra(845c2959-d054-47a6-8b05-996e01479535)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" podUID="845c2959-d054-47a6-8b05-996e01479535"
Jul 16 22:35:12.802035 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.802005    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:35:12.817888 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:12.817849    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ckb4p"]
Jul 16 22:35:13.504728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:13.504697    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:35:13.676929 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:13.676895    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" path="/var/lib/kubelet/pods/49b2c5e5-df18-4768-b634-ed49d2b9c65e/volumes"
Jul 16 22:35:19.341526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.341483    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:19.342021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342002    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342026    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342041    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342050    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342255 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342144    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342255 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342157    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.342255 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.342167    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:19.346596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.346572    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:19.362242 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.362213    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:19.386081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.386045    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n\" (UID: \"4490060d-fce3-466d-abb9-e9df43cb6286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:19.487295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.487256    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n\" (UID: \"4490060d-fce3-466d-abb9-e9df43cb6286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:19.487942 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.487921    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n\" (UID: \"4490060d-fce3-466d-abb9-e9df43cb6286\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:19.657225 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.657130    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:19.803102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:19.803080    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:19.805581 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:35:19.805558    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4490060d_fce3_466d_abb9_e9df43cb6286.slice/crio-5a220751ed0fb626770420d4045f9cc80afd897b06e9f6daa494f1ed107826e2 WatchSource:0}: Error finding container 5a220751ed0fb626770420d4045f9cc80afd897b06e9f6daa494f1ed107826e2: Status 404 returned error can't find the container with id 5a220751ed0fb626770420d4045f9cc80afd897b06e9f6daa494f1ed107826e2
Jul 16 22:35:20.801505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:20.801442    2555 generic.go:356] "Generic (PLEG): container finished" podID="4490060d-fce3-466d-abb9-e9df43cb6286" containerID="3a84b9a8c09633676bb24ea6f7f52c795410c00219431da56b723aa035852e91" exitCode=6
Jul 16 22:35:20.801505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:20.801492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" event={"ID":"4490060d-fce3-466d-abb9-e9df43cb6286","Type":"ContainerDied","Data":"3a84b9a8c09633676bb24ea6f7f52c795410c00219431da56b723aa035852e91"}
Jul 16 22:35:20.801928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:20.801544    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" event={"ID":"4490060d-fce3-466d-abb9-e9df43cb6286","Type":"ContainerStarted","Data":"5a220751ed0fb626770420d4045f9cc80afd897b06e9f6daa494f1ed107826e2"}
Jul 16 22:35:20.801928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:20.801734    2555 scope.go:117] "RemoveContainer" containerID="3a84b9a8c09633676bb24ea6f7f52c795410c00219431da56b723aa035852e91"
Jul 16 22:35:21.808210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:21.808165    2555 generic.go:356] "Generic (PLEG): container finished" podID="4490060d-fce3-466d-abb9-e9df43cb6286" containerID="f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6" exitCode=6
Jul 16 22:35:21.808661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:21.808285    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" event={"ID":"4490060d-fce3-466d-abb9-e9df43cb6286","Type":"ContainerDied","Data":"f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6"}
Jul 16 22:35:21.808661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:21.808336    2555 scope.go:117] "RemoveContainer" containerID="3a84b9a8c09633676bb24ea6f7f52c795410c00219431da56b723aa035852e91"
Jul 16 22:35:21.808661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:21.808440    2555 scope.go:117] "RemoveContainer" containerID="f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6"
Jul 16 22:35:21.808803 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:21.808687    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n_odh-ai-gateway-infra(4490060d-fce3-466d-abb9-e9df43cb6286)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" podUID="4490060d-fce3-466d-abb9-e9df43cb6286"
Jul 16 22:35:22.815705 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:22.815678    2555 scope.go:117] "RemoveContainer" containerID="f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6"
Jul 16 22:35:22.816056 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:22.815888    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n_odh-ai-gateway-infra(4490060d-fce3-466d-abb9-e9df43cb6286)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" podUID="4490060d-fce3-466d-abb9-e9df43cb6286"
Jul 16 22:35:24.671035 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:24.671005    2555 scope.go:117] "RemoveContainer" containerID="d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99"
Jul 16 22:35:25.833140 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:25.833107    2555 generic.go:356] "Generic (PLEG): container finished" podID="845c2959-d054-47a6-8b05-996e01479535" containerID="e5d83d6e51352762fb8e26883f5ed6b142eabd98480cff48c9824bcc98aa4d59" exitCode=6
Jul 16 22:35:25.833586 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:25.833177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" event={"ID":"845c2959-d054-47a6-8b05-996e01479535","Type":"ContainerDied","Data":"e5d83d6e51352762fb8e26883f5ed6b142eabd98480cff48c9824bcc98aa4d59"}
Jul 16 22:35:25.833586 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:25.833239    2555 scope.go:117] "RemoveContainer" containerID="d2b3e0b0d69a3f09c6d92aec82d1d5ae5df5c56b3d0457a7e2e0c5eb0f2fce99"
Jul 16 22:35:25.833586 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:25.833520    2555 scope.go:117] "RemoveContainer" containerID="e5d83d6e51352762fb8e26883f5ed6b142eabd98480cff48c9824bcc98aa4d59"
Jul 16 22:35:25.833761 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:25.833737    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq_odh-ai-gateway-infra(845c2959-d054-47a6-8b05-996e01479535)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" podUID="845c2959-d054-47a6-8b05-996e01479535"
Jul 16 22:35:26.869745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:26.869716    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:27.010250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.010226    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:27.155786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.155683    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca\") pod \"845c2959-d054-47a6-8b05-996e01479535\" (UID: \"845c2959-d054-47a6-8b05-996e01479535\") "
Jul 16 22:35:27.156080 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.156058    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "845c2959-d054-47a6-8b05-996e01479535" (UID: "845c2959-d054-47a6-8b05-996e01479535"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:27.256543 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.256504    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/845c2959-d054-47a6-8b05-996e01479535-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:35:27.848091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.848056    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq" event={"ID":"845c2959-d054-47a6-8b05-996e01479535","Type":"ContainerDied","Data":"7d77984f96884f9a9c5c33ec04c50728bf80334ed1eb932a58a4f0cfa7c3b462"}
Jul 16 22:35:27.848091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.848084    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"
Jul 16 22:35:27.848350 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.848116    2555 scope.go:117] "RemoveContainer" containerID="e5d83d6e51352762fb8e26883f5ed6b142eabd98480cff48c9824bcc98aa4d59"
Jul 16 22:35:27.865649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.865627    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:27.870421 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:27.870397    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-s65tq"]
Jul 16 22:35:29.676518 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:29.676485    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="845c2959-d054-47a6-8b05-996e01479535" path="/var/lib/kubelet/pods/845c2959-d054-47a6-8b05-996e01479535/volumes"
Jul 16 22:35:30.005964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.005871    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:35:30.550640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550607    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:30.550958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550946    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550959    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550971    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550977    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="49b2c5e5-df18-4768-b634-ed49d2b9c65e" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550986    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550992    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.550998    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.551003    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551245 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.551061    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551245 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.551068    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.551245 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.551075    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="845c2959-d054-47a6-8b05-996e01479535" containerName="revoke-keys"
Jul 16 22:35:30.555271 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.555253    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:30.568692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.568667    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:30.689980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.689940    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck\" (UID: \"2b42eda9-11ec-4708-b70b-fb113bd2fed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:30.791491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.791436    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck\" (UID: \"2b42eda9-11ec-4708-b70b-fb113bd2fed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:30.792094 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.792075    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck\" (UID: \"2b42eda9-11ec-4708-b70b-fb113bd2fed6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:30.865071 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:30.864982    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:31.018391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:31.018368    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:31.020318 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:35:31.020291    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b42eda9_11ec_4708_b70b_fb113bd2fed6.slice/crio-3291b9b11f848ddddc9f1ec5323330e0a9f7cad0e544f23dce9fa1a7a0e59d15 WatchSource:0}: Error finding container 3291b9b11f848ddddc9f1ec5323330e0a9f7cad0e544f23dce9fa1a7a0e59d15: Status 404 returned error can't find the container with id 3291b9b11f848ddddc9f1ec5323330e0a9f7cad0e544f23dce9fa1a7a0e59d15
Jul 16 22:35:31.870182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:31.870097    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerID="be9a6286245e95ec9b4efcc5ec6252ded77f933ca748a39af8e74699196a0ee1" exitCode=6
Jul 16 22:35:31.870642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:31.870184    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" event={"ID":"2b42eda9-11ec-4708-b70b-fb113bd2fed6","Type":"ContainerDied","Data":"be9a6286245e95ec9b4efcc5ec6252ded77f933ca748a39af8e74699196a0ee1"}
Jul 16 22:35:31.870642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:31.870237    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" event={"ID":"2b42eda9-11ec-4708-b70b-fb113bd2fed6","Type":"ContainerStarted","Data":"3291b9b11f848ddddc9f1ec5323330e0a9f7cad0e544f23dce9fa1a7a0e59d15"}
Jul 16 22:35:31.870642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:31.870315    2555 scope.go:117] "RemoveContainer" containerID="be9a6286245e95ec9b4efcc5ec6252ded77f933ca748a39af8e74699196a0ee1"
Jul 16 22:35:32.877928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:32.877891    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerID="c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05" exitCode=6
Jul 16 22:35:32.878396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:32.877974    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" event={"ID":"2b42eda9-11ec-4708-b70b-fb113bd2fed6","Type":"ContainerDied","Data":"c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05"}
Jul 16 22:35:32.878396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:32.878031    2555 scope.go:117] "RemoveContainer" containerID="be9a6286245e95ec9b4efcc5ec6252ded77f933ca748a39af8e74699196a0ee1"
Jul 16 22:35:32.878396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:32.878284    2555 scope.go:117] "RemoveContainer" containerID="c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05"
Jul 16 22:35:32.878544 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:32.878524    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck_odh-ai-gateway-infra(2b42eda9-11ec-4708-b70b-fb113bd2fed6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6"
Jul 16 22:35:33.670981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:33.670948    2555 scope.go:117] "RemoveContainer" containerID="f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6"
Jul 16 22:35:33.885608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:33.885579    2555 scope.go:117] "RemoveContainer" containerID="c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05"
Jul 16 22:35:33.885976 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:33.885772    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck_odh-ai-gateway-infra(2b42eda9-11ec-4708-b70b-fb113bd2fed6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6"
Jul 16 22:35:34.892582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:34.892545    2555 generic.go:356] "Generic (PLEG): container finished" podID="4490060d-fce3-466d-abb9-e9df43cb6286" containerID="66a3ac981120b20cd44b70aeef369b6b6789cefb18601d47e087c1d9faf41626" exitCode=6
Jul 16 22:35:34.893033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:34.892595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" event={"ID":"4490060d-fce3-466d-abb9-e9df43cb6286","Type":"ContainerDied","Data":"66a3ac981120b20cd44b70aeef369b6b6789cefb18601d47e087c1d9faf41626"}
Jul 16 22:35:34.893033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:34.892672    2555 scope.go:117] "RemoveContainer" containerID="f396620435e36c501169d8be709c2dc091da47aa2adddd40e93a302140743ca6"
Jul 16 22:35:34.893033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:34.892976    2555 scope.go:117] "RemoveContainer" containerID="66a3ac981120b20cd44b70aeef369b6b6789cefb18601d47e087c1d9faf41626"
Jul 16 22:35:34.893234 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:34.893215    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n_odh-ai-gateway-infra(4490060d-fce3-466d-abb9-e9df43cb6286)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" podUID="4490060d-fce3-466d-abb9-e9df43cb6286"
Jul 16 22:35:35.924091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:35.924055    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:36.070002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.069979    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:36.139867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.139836    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca\") pod \"4490060d-fce3-466d-abb9-e9df43cb6286\" (UID: \"4490060d-fce3-466d-abb9-e9df43cb6286\") "
Jul 16 22:35:36.140224 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.140199    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4490060d-fce3-466d-abb9-e9df43cb6286" (UID: "4490060d-fce3-466d-abb9-e9df43cb6286"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:36.240914 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.240828    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4490060d-fce3-466d-abb9-e9df43cb6286-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:35:36.906516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.906481    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n" event={"ID":"4490060d-fce3-466d-abb9-e9df43cb6286","Type":"ContainerDied","Data":"5a220751ed0fb626770420d4045f9cc80afd897b06e9f6daa494f1ed107826e2"}
Jul 16 22:35:36.906516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.906516    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"
Jul 16 22:35:36.906754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.906544    2555 scope.go:117] "RemoveContainer" containerID="66a3ac981120b20cd44b70aeef369b6b6789cefb18601d47e087c1d9faf41626"
Jul 16 22:35:36.929199 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.929165    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:36.936092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:36.936068    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-l7k8n"]
Jul 16 22:35:37.676232 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:37.676201    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" path="/var/lib/kubelet/pods/4490060d-fce3-466d-abb9-e9df43cb6286/volumes"
Jul 16 22:35:41.684692 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:41.684661    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:35:44.670592 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:44.670560    2555 scope.go:117] "RemoveContainer" containerID="c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05"
Jul 16 22:35:45.958804 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:45.958766    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerID="bb5c7d082d948db9101f54d999684c6c6ef7542ad7d8610201e0d02d838dda22" exitCode=6
Jul 16 22:35:45.959293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:45.958832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" event={"ID":"2b42eda9-11ec-4708-b70b-fb113bd2fed6","Type":"ContainerDied","Data":"bb5c7d082d948db9101f54d999684c6c6ef7542ad7d8610201e0d02d838dda22"}
Jul 16 22:35:45.959293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:45.958897    2555 scope.go:117] "RemoveContainer" containerID="c2deefb86a47d635c7c4cc01b12f1459a35919ea5d3f4069c141d2950f9f1c05"
Jul 16 22:35:45.959293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:45.959234    2555 scope.go:117] "RemoveContainer" containerID="bb5c7d082d948db9101f54d999684c6c6ef7542ad7d8610201e0d02d838dda22"
Jul 16 22:35:45.959515 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:45.959486    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck_odh-ai-gateway-infra(2b42eda9-11ec-4708-b70b-fb113bd2fed6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6"
Jul 16 22:35:46.998096 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:46.998062    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:47.128209 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.128186    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:47.245500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.245431    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca\") pod \"2b42eda9-11ec-4708-b70b-fb113bd2fed6\" (UID: \"2b42eda9-11ec-4708-b70b-fb113bd2fed6\") "
Jul 16 22:35:47.245800 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.245776    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b42eda9-11ec-4708-b70b-fb113bd2fed6" (UID: "2b42eda9-11ec-4708-b70b-fb113bd2fed6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:35:47.346536 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.346495    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b42eda9-11ec-4708-b70b-fb113bd2fed6-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:35:47.972424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.972336    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck" event={"ID":"2b42eda9-11ec-4708-b70b-fb113bd2fed6","Type":"ContainerDied","Data":"3291b9b11f848ddddc9f1ec5323330e0a9f7cad0e544f23dce9fa1a7a0e59d15"}
Jul 16 22:35:47.972424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.972353    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"
Jul 16 22:35:47.972424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.972404    2555 scope.go:117] "RemoveContainer" containerID="bb5c7d082d948db9101f54d999684c6c6ef7542ad7d8610201e0d02d838dda22"
Jul 16 22:35:47.989721 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.989699    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:47.993927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:47.993907    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-z5gck"]
Jul 16 22:35:49.675740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:49.675706    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" path="/var/lib/kubelet/pods/2b42eda9-11ec-4708-b70b-fb113bd2fed6/volumes"
Jul 16 22:35:51.115478 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115434    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:35:51.115902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115884    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.115958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115905    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.115958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115915    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.115958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115922    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115935    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.115985    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116002    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116007    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116061    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116068    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116076    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.116332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116081    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.116332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.116086    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.120386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.120368    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.125086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.125064    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2dbd48\"/\"kube-root-ca.crt\""
Jul 16 22:35:51.126036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.126015    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2dbd48\"/\"test-model-2dbd48-kserve-self-signed-certs\""
Jul 16 22:35:51.126158 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.126073    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-2dbd48\"/\"openshift-service-ca.crt\""
Jul 16 22:35:51.126242 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.126220    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-2dbd48\"/\"default-dockercfg-q2lx2\""
Jul 16 22:35:51.135734 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.135710    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:35:51.176167 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176137    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.176310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176172    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.176310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176287    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.176395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176314    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.176395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176336    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.176496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.176408    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm8hw\" (UniqueName: \"kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.191815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.191772    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:35:51.192680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.192657    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.192680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.192681    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="4490060d-fce3-466d-abb9-e9df43cb6286" containerName="revoke-keys"
Jul 16 22:35:51.192912 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.192738    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.192912 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.192747    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.193051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.192921    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b42eda9-11ec-4708-b70b-fb113bd2fed6" containerName="revoke-keys"
Jul 16 22:35:51.197859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.197831    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:35:51.200928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.200904    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:35:51.201313 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.201122    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:35:51.203182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.203125    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:35:51.277515 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277478    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.277515 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277516    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.277753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277538    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tbk7v\" (UID: \"5a8f9109-5bc1-497c-afb4-c6a79150ed3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:35:51.277753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.277753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277642    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.277753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277684    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.277753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277742    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm8hw\" (UniqueName: \"kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.278011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.277989    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.278067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.278044    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.278112 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.278081    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.280088 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.280045    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.280585 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.280567    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.287097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.287063    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm8hw\" (UniqueName: \"kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw\") pod \"test-model-2dbd48-kserve-57d749cc49-8hk7t\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") " pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.378307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.378218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tbk7v\" (UID: \"5a8f9109-5bc1-497c-afb4-c6a79150ed3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:35:51.378828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.378811    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-tbk7v\" (UID: \"5a8f9109-5bc1-497c-afb4-c6a79150ed3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:35:51.431483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.431430    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:35:51.510916 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.510888    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:35:51.573918 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.573554    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:35:51.657702 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.657662    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:35:51.661255 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:35:51.661218    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a8f9109_5bc1_497c_afb4_c6a79150ed3b.slice/crio-1b505989580d2f6ef3d972f588fe0d24108f5ceaf676f78839c4a8b318b77dd2 WatchSource:0}: Error finding container 1b505989580d2f6ef3d972f588fe0d24108f5ceaf676f78839c4a8b318b77dd2: Status 404 returned error can't find the container with id 1b505989580d2f6ef3d972f588fe0d24108f5ceaf676f78839c4a8b318b77dd2
Jul 16 22:35:51.996069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.996027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" event={"ID":"1d1ea759-df06-4cd5-a183-4e456ddae446","Type":"ContainerStarted","Data":"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"}
Jul 16 22:35:51.996069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.996073    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" event={"ID":"1d1ea759-df06-4cd5-a183-4e456ddae446","Type":"ContainerStarted","Data":"fec1872dfb8be7524c8e6b99308f7e67fdce4eed5bf0472991e342ac8687a0ad"}
Jul 16 22:35:51.997621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.997594    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerStarted","Data":"7f242b7b3f754d4792b0bc8290eb04f27297207cf64da43d03fd26774582e607"}
Jul 16 22:35:51.997732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:51.997640    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerStarted","Data":"1b505989580d2f6ef3d972f588fe0d24108f5ceaf676f78839c4a8b318b77dd2"}
Jul 16 22:35:52.014435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:52.014394    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" podStartSLOduration=1.014378364 podStartE2EDuration="1.014378364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:52.01342656 +0000 UTC m=+2806.850480362" watchObservedRunningTime="2026-07-16 22:35:52.014378364 +0000 UTC m=+2806.851432164"
Jul 16 22:35:52.030121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:52.030062    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" podStartSLOduration=1.030042838 podStartE2EDuration="1.030042838s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:35:52.02994614 +0000 UTC m=+2806.866999941" watchObservedRunningTime="2026-07-16 22:35:52.030042838 +0000 UTC m=+2806.867096639"
Jul 16 22:35:53.004323 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:53.004287    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerID="7f242b7b3f754d4792b0bc8290eb04f27297207cf64da43d03fd26774582e607" exitCode=6
Jul 16 22:35:53.004717 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:53.004377    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerDied","Data":"7f242b7b3f754d4792b0bc8290eb04f27297207cf64da43d03fd26774582e607"}
Jul 16 22:35:53.004717 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:53.004552    2555 scope.go:117] "RemoveContainer" containerID="7f242b7b3f754d4792b0bc8290eb04f27297207cf64da43d03fd26774582e607"
Jul 16 22:35:54.011733 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:54.011630    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerID="5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98" exitCode=6
Jul 16 22:35:54.011733 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:54.011671    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerDied","Data":"5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98"}
Jul 16 22:35:54.011733 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:54.011732    2555 scope.go:117] "RemoveContainer" containerID="7f242b7b3f754d4792b0bc8290eb04f27297207cf64da43d03fd26774582e607"
Jul 16 22:35:54.012244 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:54.011903    2555 scope.go:117] "RemoveContainer" containerID="5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98"
Jul 16 22:35:54.012244 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:54.012120    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tbk7v_odh-ai-gateway-infra(5a8f9109-5bc1-497c-afb4-c6a79150ed3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b"
Jul 16 22:35:55.018725 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:55.018700    2555 scope.go:117] "RemoveContainer" containerID="5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98"
Jul 16 22:35:55.019094 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:35:55.018910    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tbk7v_odh-ai-gateway-infra(5a8f9109-5bc1-497c-afb4-c6a79150ed3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b"
Jul 16 22:35:55.993503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:55.993454    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:35:59.255938 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.255904    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:35:59.260569 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.260553    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:35:59.267124 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.267098    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:35:59.338429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.338393    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng\" (UID: \"aeba430d-a023-451a-bcf7-5ca23e1c9fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:35:59.439413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.439379    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng\" (UID: \"aeba430d-a023-451a-bcf7-5ca23e1c9fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:35:59.440024 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.440004    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng\" (UID: \"aeba430d-a023-451a-bcf7-5ca23e1c9fb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:35:59.573609 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.573572    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:35:59.715560 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:35:59.715534    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:35:59.717537 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:35:59.717509    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeba430d_a023_451a_bcf7_5ca23e1c9fb8.slice/crio-d3827c3c7c8e7f95c0bc607899e4f6acc872f4e458eef5c36cd90fb3b9ab90ed WatchSource:0}: Error finding container d3827c3c7c8e7f95c0bc607899e4f6acc872f4e458eef5c36cd90fb3b9ab90ed: Status 404 returned error can't find the container with id d3827c3c7c8e7f95c0bc607899e4f6acc872f4e458eef5c36cd90fb3b9ab90ed
Jul 16 22:36:00.051249 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:00.051212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerStarted","Data":"c2634c883e483e796bcc7d8df7497f525500ec766d1deae7857d971b2fc5a0a1"}
Jul 16 22:36:00.051485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:00.051267    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerStarted","Data":"d3827c3c7c8e7f95c0bc607899e4f6acc872f4e458eef5c36cd90fb3b9ab90ed"}
Jul 16 22:36:00.070330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:00.070272    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" podStartSLOduration=1.070254313 podStartE2EDuration="1.070254313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:00.069870582 +0000 UTC m=+2814.906924398" watchObservedRunningTime="2026-07-16 22:36:00.070254313 +0000 UTC m=+2814.907308137"
Jul 16 22:36:01.059327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.059287    2555 generic.go:356] "Generic (PLEG): container finished" podID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerID="c2634c883e483e796bcc7d8df7497f525500ec766d1deae7857d971b2fc5a0a1" exitCode=6
Jul 16 22:36:01.059744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.059332    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerDied","Data":"c2634c883e483e796bcc7d8df7497f525500ec766d1deae7857d971b2fc5a0a1"}
Jul 16 22:36:01.059744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.059599    2555 scope.go:117] "RemoveContainer" containerID="c2634c883e483e796bcc7d8df7497f525500ec766d1deae7857d971b2fc5a0a1"
Jul 16 22:36:01.432343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.432304    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:01.432549 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.432362    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:01.445576 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:01.445553    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:02.067097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:02.067055    2555 generic.go:356] "Generic (PLEG): container finished" podID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerID="4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7" exitCode=6
Jul 16 22:36:02.067603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:02.067136    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerDied","Data":"4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7"}
Jul 16 22:36:02.067603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:02.067202    2555 scope.go:117] "RemoveContainer" containerID="c2634c883e483e796bcc7d8df7497f525500ec766d1deae7857d971b2fc5a0a1"
Jul 16 22:36:02.067603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:02.067437    2555 scope.go:117] "RemoveContainer" containerID="4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7"
Jul 16 22:36:02.067785 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:02.067710    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng_odh-ai-gateway-infra(aeba430d-a023-451a-bcf7-5ca23e1c9fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8"
Jul 16 22:36:02.078986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:02.078963    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:03.074795 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:03.074760    2555 scope.go:117] "RemoveContainer" containerID="4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7"
Jul 16 22:36:03.075192 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:03.074946    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng_odh-ai-gateway-infra(aeba430d-a023-451a-bcf7-5ca23e1c9fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8"
Jul 16 22:36:09.379189 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.379151    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:09.384273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.384256    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:09.391482 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.391421    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:09.430699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.430668    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll\" (UID: \"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:09.532135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.532102    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll\" (UID: \"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:09.532755 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.532738    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll\" (UID: \"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:09.670722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.670629    2555 scope.go:117] "RemoveContainer" containerID="5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98"
Jul 16 22:36:09.696669 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.696645    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:09.850678 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:09.850652    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:09.852373 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:09.852334    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272b15c2_b21d_4c7c_b4eb_ae9e5dbeb2c5.slice/crio-88c4166f7d875d2e0e9f297e8c7e38b06268e901b8a87aec55bb4e42ca8560e5 WatchSource:0}: Error finding container 88c4166f7d875d2e0e9f297e8c7e38b06268e901b8a87aec55bb4e42ca8560e5: Status 404 returned error can't find the container with id 88c4166f7d875d2e0e9f297e8c7e38b06268e901b8a87aec55bb4e42ca8560e5
Jul 16 22:36:10.121531 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:10.121492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerStarted","Data":"08baca6e4268e7aa297a332c307772b55ec3e4027afa7f35923e66366f78524d"}
Jul 16 22:36:10.122912 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:10.122891    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerStarted","Data":"66f2dcd831bc209ad037691f1cf24c3b4e742d94622e26e35d901ce833be5b4c"}
Jul 16 22:36:10.123027 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:10.122928    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerStarted","Data":"88c4166f7d875d2e0e9f297e8c7e38b06268e901b8a87aec55bb4e42ca8560e5"}
Jul 16 22:36:10.155069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:10.155016    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" podStartSLOduration=1.154998885 podStartE2EDuration="1.154998885s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:10.152986819 +0000 UTC m=+2824.990040641" watchObservedRunningTime="2026-07-16 22:36:10.154998885 +0000 UTC m=+2824.992052685"
Jul 16 22:36:10.708794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:10.708762    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:36:11.130738 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.130706    2555 generic.go:356] "Generic (PLEG): container finished" podID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerID="08baca6e4268e7aa297a332c307772b55ec3e4027afa7f35923e66366f78524d" exitCode=6
Jul 16 22:36:11.130903 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.130778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerDied","Data":"08baca6e4268e7aa297a332c307772b55ec3e4027afa7f35923e66366f78524d"}
Jul 16 22:36:11.130903 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.130845    2555 scope.go:117] "RemoveContainer" containerID="5514ba93847186cfa654fc672079bd90cadde3557ee407c872f9e3ec751a9d98"
Jul 16 22:36:11.131154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.131140    2555 scope.go:117] "RemoveContainer" containerID="08baca6e4268e7aa297a332c307772b55ec3e4027afa7f35923e66366f78524d"
Jul 16 22:36:11.131366 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:11.131348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-tbk7v_odh-ai-gateway-infra(5a8f9109-5bc1-497c-afb4-c6a79150ed3b)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b"
Jul 16 22:36:11.132505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.132488    2555 generic.go:356] "Generic (PLEG): container finished" podID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerID="66f2dcd831bc209ad037691f1cf24c3b4e742d94622e26e35d901ce833be5b4c" exitCode=6
Jul 16 22:36:11.132585 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.132573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerDied","Data":"66f2dcd831bc209ad037691f1cf24c3b4e742d94622e26e35d901ce833be5b4c"}
Jul 16 22:36:11.132775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.132762    2555 scope.go:117] "RemoveContainer" containerID="66f2dcd831bc209ad037691f1cf24c3b4e742d94622e26e35d901ce833be5b4c"
Jul 16 22:36:11.149741 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:11.149719    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:36:12.141709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.141681    2555 generic.go:356] "Generic (PLEG): container finished" podID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerID="73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13" exitCode=6
Jul 16 22:36:12.142064 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.141762    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerDied","Data":"73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13"}
Jul 16 22:36:12.142064 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.141838    2555 scope.go:117] "RemoveContainer" containerID="66f2dcd831bc209ad037691f1cf24c3b4e742d94622e26e35d901ce833be5b4c"
Jul 16 22:36:12.142064 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.141970    2555 scope.go:117] "RemoveContainer" containerID="73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13"
Jul 16 22:36:12.142227 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:12.142201    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll_odh-ai-gateway-infra(272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5"
Jul 16 22:36:12.279106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.279073    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:36:12.360865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.360827    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca\") pod \"5a8f9109-5bc1-497c-afb4-c6a79150ed3b\" (UID: \"5a8f9109-5bc1-497c-afb4-c6a79150ed3b\") "
Jul 16 22:36:12.361231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.361210    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5a8f9109-5bc1-497c-afb4-c6a79150ed3b" (UID: "5a8f9109-5bc1-497c-afb4-c6a79150ed3b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:12.461749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:12.461658    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a8f9109-5bc1-497c-afb4-c6a79150ed3b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:13.148028 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.148002    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"
Jul 16 22:36:13.148547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.148040    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v" event={"ID":"5a8f9109-5bc1-497c-afb4-c6a79150ed3b","Type":"ContainerDied","Data":"1b505989580d2f6ef3d972f588fe0d24108f5ceaf676f78839c4a8b318b77dd2"}
Jul 16 22:36:13.148547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.148112    2555 scope.go:117] "RemoveContainer" containerID="08baca6e4268e7aa297a332c307772b55ec3e4027afa7f35923e66366f78524d"
Jul 16 22:36:13.149950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.149934    2555 scope.go:117] "RemoveContainer" containerID="73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13"
Jul 16 22:36:13.150129 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:13.150112    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll_odh-ai-gateway-infra(272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5"
Jul 16 22:36:13.287949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.287915    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:36:13.291975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.291949    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-tbk7v"]
Jul 16 22:36:13.676772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:13.676739    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" path="/var/lib/kubelet/pods/5a8f9109-5bc1-497c-afb4-c6a79150ed3b/volumes"
Jul 16 22:36:17.670487 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.670428    2555 scope.go:117] "RemoveContainer" containerID="4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7"
Jul 16 22:36:17.773056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.773026    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:36:17.908386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908350    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"]
Jul 16 22:36:17.908874 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908854    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.908874 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908875    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.908998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908893    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.908998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908898    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.908998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908983    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.908998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.908991    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:17.911898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.911818    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:17.919292 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:17.919260    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"]
Jul 16 22:36:18.009880 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.009790    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb\" (UID: \"004d3af7-5f6e-49de-ae4f-f881af25ca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:18.111123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.111086    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb\" (UID: \"004d3af7-5f6e-49de-ae4f-f881af25ca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:18.111709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.111692    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb\" (UID: \"004d3af7-5f6e-49de-ae4f-f881af25ca77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:18.179706 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.179665    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerStarted","Data":"f0ad8788dc06ccfe9c03af04fd04c003f5877c2ad42713129a21bfc6553fcfc7"}
Jul 16 22:36:18.224036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.224006    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:18.364542 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:18.364513    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"]
Jul 16 22:36:18.367308 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:18.366726    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod004d3af7_5f6e_49de_ae4f_f881af25ca77.slice/crio-b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6 WatchSource:0}: Error finding container b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6: Status 404 returned error can't find the container with id b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6
Jul 16 22:36:19.187829 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.187792    2555 generic.go:356] "Generic (PLEG): container finished" podID="004d3af7-5f6e-49de-ae4f-f881af25ca77" containerID="fd00fa6625f528f30b6186fd724a3952fa8138174d39c1f77bc4fa090e2b426e" exitCode=0
Jul 16 22:36:19.188265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.187880    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb" event={"ID":"004d3af7-5f6e-49de-ae4f-f881af25ca77","Type":"ContainerDied","Data":"fd00fa6625f528f30b6186fd724a3952fa8138174d39c1f77bc4fa090e2b426e"}
Jul 16 22:36:19.188265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.187933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb" event={"ID":"004d3af7-5f6e-49de-ae4f-f881af25ca77","Type":"ContainerStarted","Data":"b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6"}
Jul 16 22:36:19.189933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.189910    2555 generic.go:356] "Generic (PLEG): container finished" podID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerID="f0ad8788dc06ccfe9c03af04fd04c003f5877c2ad42713129a21bfc6553fcfc7" exitCode=6
Jul 16 22:36:19.190059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.189954    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerDied","Data":"f0ad8788dc06ccfe9c03af04fd04c003f5877c2ad42713129a21bfc6553fcfc7"}
Jul 16 22:36:19.190059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.189995    2555 scope.go:117] "RemoveContainer" containerID="4644ae2c89458fea6cafbf9a622b99024a1a4376732dd3c8ba711df8815380f7"
Jul 16 22:36:19.190316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.190300    2555 scope.go:117] "RemoveContainer" containerID="f0ad8788dc06ccfe9c03af04fd04c003f5877c2ad42713129a21bfc6553fcfc7"
Jul 16 22:36:19.190541 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:19.190524    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng_odh-ai-gateway-infra(aeba430d-a023-451a-bcf7-5ca23e1c9fb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8"
Jul 16 22:36:19.224669 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:19.224644    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:36:20.356962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.356936    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:20.358950 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.358933    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:36:20.433310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.433283    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca\") pod \"004d3af7-5f6e-49de-ae4f-f881af25ca77\" (UID: \"004d3af7-5f6e-49de-ae4f-f881af25ca77\") "
Jul 16 22:36:20.433522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.433354    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca\") pod \"aeba430d-a023-451a-bcf7-5ca23e1c9fb8\" (UID: \"aeba430d-a023-451a-bcf7-5ca23e1c9fb8\") "
Jul 16 22:36:20.433688 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.433652    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "004d3af7-5f6e-49de-ae4f-f881af25ca77" (UID: "004d3af7-5f6e-49de-ae4f-f881af25ca77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:20.433793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.433727    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aeba430d-a023-451a-bcf7-5ca23e1c9fb8" (UID: "aeba430d-a023-451a-bcf7-5ca23e1c9fb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:20.534359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.534329    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/004d3af7-5f6e-49de-ae4f-f881af25ca77-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:20.534359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.534359    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeba430d-a023-451a-bcf7-5ca23e1c9fb8-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:20.570801 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.570768    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:20.571126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571114    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.571166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571128    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.571166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571140    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:20.571166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571145    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:20.571166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571159    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.571166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571165    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571174    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="004d3af7-5f6e-49de-ae4f-f881af25ca77" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571180    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="004d3af7-5f6e-49de-ae4f-f881af25ca77" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571252    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="004d3af7-5f6e-49de-ae4f-f881af25ca77" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571260    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a8f9109-5bc1-497c-afb4-c6a79150ed3b" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571266    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.571316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.571272    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:20.574369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.574352    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:20.582340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.581677    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:20.635737 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.635703    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn\" (UID: \"9ed6048f-440b-493d-8740-cefc56decd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:20.736658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.736624    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn\" (UID: \"9ed6048f-440b-493d-8740-cefc56decd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:20.737183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.737166    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn\" (UID: \"9ed6048f-440b-493d-8740-cefc56decd5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:20.886324 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:20.886232    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:21.019895 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.019871    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:21.021569 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:21.021544    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ed6048f_440b_493d_8740_cefc56decd5d.slice/crio-f818a86e628abe3dcb4b16216b2ae583c51bf3885be26da5641bd2a22a94028f WatchSource:0}: Error finding container f818a86e628abe3dcb4b16216b2ae583c51bf3885be26da5641bd2a22a94028f: Status 404 returned error can't find the container with id f818a86e628abe3dcb4b16216b2ae583c51bf3885be26da5641bd2a22a94028f
Jul 16 22:36:21.204363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.204318    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" event={"ID":"9ed6048f-440b-493d-8740-cefc56decd5d","Type":"ContainerStarted","Data":"f818a86e628abe3dcb4b16216b2ae583c51bf3885be26da5641bd2a22a94028f"}
Jul 16 22:36:21.205784 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.205755    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng" event={"ID":"aeba430d-a023-451a-bcf7-5ca23e1c9fb8","Type":"ContainerDied","Data":"d3827c3c7c8e7f95c0bc607899e4f6acc872f4e458eef5c36cd90fb3b9ab90ed"}
Jul 16 22:36:21.205939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.205828    2555 scope.go:117] "RemoveContainer" containerID="f0ad8788dc06ccfe9c03af04fd04c003f5877c2ad42713129a21bfc6553fcfc7"
Jul 16 22:36:21.205939 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.205763    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"
Jul 16 22:36:21.207248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.207220    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb" event={"ID":"004d3af7-5f6e-49de-ae4f-f881af25ca77","Type":"ContainerDied","Data":"b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6"}
Jul 16 22:36:21.207337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.207265    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b37329cd1cf97d84541699f5ae691531872485c86bb8c818d198068e23168ed6"
Jul 16 22:36:21.207337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.207325    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb"
Jul 16 22:36:21.236443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.236405    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:36:21.241586 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.241559    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-hfpng"]
Jul 16 22:36:21.676051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:21.676018    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" path="/var/lib/kubelet/pods/aeba430d-a023-451a-bcf7-5ca23e1c9fb8/volumes"
Jul 16 22:36:22.214297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:22.214259    2555 generic.go:356] "Generic (PLEG): container finished" podID="9ed6048f-440b-493d-8740-cefc56decd5d" containerID="7caf8debac1b1f53687b12c9111e66090938737877c5146815c290ea00c17baa" exitCode=6
Jul 16 22:36:22.214537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:22.214350    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" event={"ID":"9ed6048f-440b-493d-8740-cefc56decd5d","Type":"ContainerDied","Data":"7caf8debac1b1f53687b12c9111e66090938737877c5146815c290ea00c17baa"}
Jul 16 22:36:22.214612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:22.214578    2555 scope.go:117] "RemoveContainer" containerID="7caf8debac1b1f53687b12c9111e66090938737877c5146815c290ea00c17baa"
Jul 16 22:36:23.222608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:23.222511    2555 generic.go:356] "Generic (PLEG): container finished" podID="9ed6048f-440b-493d-8740-cefc56decd5d" containerID="7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8" exitCode=6
Jul 16 22:36:23.222608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:23.222595    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" event={"ID":"9ed6048f-440b-493d-8740-cefc56decd5d","Type":"ContainerDied","Data":"7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8"}
Jul 16 22:36:23.223110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:23.222650    2555 scope.go:117] "RemoveContainer" containerID="7caf8debac1b1f53687b12c9111e66090938737877c5146815c290ea00c17baa"
Jul 16 22:36:23.223110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:23.222867    2555 scope.go:117] "RemoveContainer" containerID="7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8"
Jul 16 22:36:23.223110 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:23.223086    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn_odh-ai-gateway-infra(9ed6048f-440b-493d-8740-cefc56decd5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" podUID="9ed6048f-440b-493d-8740-cefc56decd5d"
Jul 16 22:36:24.088389 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:24.088343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:36:24.229537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:24.229508    2555 scope.go:117] "RemoveContainer" containerID="7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8"
Jul 16 22:36:24.229927 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:24.229702    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn_odh-ai-gateway-infra(9ed6048f-440b-493d-8740-cefc56decd5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" podUID="9ed6048f-440b-493d-8740-cefc56decd5d"
Jul 16 22:36:24.670595 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:24.670554    2555 scope.go:117] "RemoveContainer" containerID="73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13"
Jul 16 22:36:25.236074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:25.236045    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerStarted","Data":"213b5f50b35b92fff275df6de07c92fae12f0fb079ffc7514a0c9ac64f71a034"}
Jul 16 22:36:26.243074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:26.243040    2555 generic.go:356] "Generic (PLEG): container finished" podID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerID="213b5f50b35b92fff275df6de07c92fae12f0fb079ffc7514a0c9ac64f71a034" exitCode=6
Jul 16 22:36:26.243562 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:26.243118    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerDied","Data":"213b5f50b35b92fff275df6de07c92fae12f0fb079ffc7514a0c9ac64f71a034"}
Jul 16 22:36:26.243562 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:26.243191    2555 scope.go:117] "RemoveContainer" containerID="73fa3397b130ae3dc85d7a46ae88d5395d76d1d179ada360712bf58e43674e13"
Jul 16 22:36:26.243562 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:26.243510    2555 scope.go:117] "RemoveContainer" containerID="213b5f50b35b92fff275df6de07c92fae12f0fb079ffc7514a0c9ac64f71a034"
Jul 16 22:36:26.243766 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:26.243742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll_odh-ai-gateway-infra(272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5"
Jul 16 22:36:26.272570 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:26.272546    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:27.388087 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:27.388060    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:27.501792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:27.501706    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca\") pod \"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5\" (UID: \"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5\") "
Jul 16 22:36:27.502096 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:27.502075    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" (UID: "272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:27.602550 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:27.602518    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:28.256732 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:28.256702    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"
Jul 16 22:36:28.256925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:28.256696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll" event={"ID":"272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5","Type":"ContainerDied","Data":"88c4166f7d875d2e0e9f297e8c7e38b06268e901b8a87aec55bb4e42ca8560e5"}
Jul 16 22:36:28.256925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:28.256841    2555 scope.go:117] "RemoveContainer" containerID="213b5f50b35b92fff275df6de07c92fae12f0fb079ffc7514a0c9ac64f71a034"
Jul 16 22:36:28.274641 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:28.274616    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:28.283794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:28.283770    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-5t5ll"]
Jul 16 22:36:29.675986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:29.675952    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" path="/var/lib/kubelet/pods/272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5/volumes"
Jul 16 22:36:32.794034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:32.793980    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" podUID="1d1ea759-df06-4cd5-a183-4e456ddae446" containerName="main" containerID="cri-o://86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583" gracePeriod=45
Jul 16 22:36:33.053115 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.053056    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:33.159859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.159823    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.159859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.159868    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pm8hw\" (UniqueName: \"kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.160120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.159891    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.160120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.159912    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.160120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.159967    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.160120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.160006    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir\") pod \"1d1ea759-df06-4cd5-a183-4e456ddae446\" (UID: \"1d1ea759-df06-4cd5-a183-4e456ddae446\") "
Jul 16 22:36:33.160334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.160188    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:33.160334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.160192    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home" (OuterVolumeSpecName: "home") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:33.160429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.160316    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:33.162297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.162271    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw" (OuterVolumeSpecName: "kube-api-access-pm8hw") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "kube-api-access-pm8hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:36:33.162412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.162308    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm" (OuterVolumeSpecName: "dshm") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:36:33.162412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.162345    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d1ea759-df06-4cd5-a183-4e456ddae446" (UID: "1d1ea759-df06-4cd5-a183-4e456ddae446"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:36:33.261708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261668    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d1ea759-df06-4cd5-a183-4e456ddae446-tls-certs\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.261708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261702    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-tmp-dir\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.261708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261713    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-model-cache\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.261925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261727    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pm8hw\" (UniqueName: \"kubernetes.io/projected/1d1ea759-df06-4cd5-a183-4e456ddae446-kube-api-access-pm8hw\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.261925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261740    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-home\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.261925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.261751    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d1ea759-df06-4cd5-a183-4e456ddae446-dshm\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:33.285821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.285790    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d1ea759-df06-4cd5-a183-4e456ddae446" containerID="86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583" exitCode=0
Jul 16 22:36:33.285973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.285834    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"
Jul 16 22:36:33.285973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.285872    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" event={"ID":"1d1ea759-df06-4cd5-a183-4e456ddae446","Type":"ContainerDied","Data":"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"}
Jul 16 22:36:33.285973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.285920    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t" event={"ID":"1d1ea759-df06-4cd5-a183-4e456ddae446","Type":"ContainerDied","Data":"fec1872dfb8be7524c8e6b99308f7e67fdce4eed5bf0472991e342ac8687a0ad"}
Jul 16 22:36:33.285973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.285941    2555 scope.go:117] "RemoveContainer" containerID="86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"
Jul 16 22:36:33.294977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.294959    2555 scope.go:117] "RemoveContainer" containerID="86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"
Jul 16 22:36:33.295224 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:33.295204    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583\": container with ID starting with 86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583 not found: ID does not exist" containerID="86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"
Jul 16 22:36:33.295290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.295236    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583"} err="failed to get container status \"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583\": rpc error: code = NotFound desc = could not find container \"86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583\": container with ID starting with 86f04539858e2d777365839abe6d058baa5a4c4850fafb8195c6a6f866a57583 not found: ID does not exist"
Jul 16 22:36:33.311977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.311924    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:36:33.318862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.318838    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-2dbd48/test-model-2dbd48-kserve-57d749cc49-8hk7t"]
Jul 16 22:36:33.609548 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.609456    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:36:33.677119 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:33.677084    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d1ea759-df06-4cd5-a183-4e456ddae446" path="/var/lib/kubelet/pods/1d1ea759-df06-4cd5-a183-4e456ddae446/volumes"
Jul 16 22:36:39.671232 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:39.671194    2555 scope.go:117] "RemoveContainer" containerID="7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8"
Jul 16 22:36:40.325321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:40.325296    2555 generic.go:356] "Generic (PLEG): container finished" podID="9ed6048f-440b-493d-8740-cefc56decd5d" containerID="1a9725d397b304bf017660bada6718b912b9cb3494b0ac8f2147e4750430d9a3" exitCode=6
Jul 16 22:36:40.325435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:40.325337    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" event={"ID":"9ed6048f-440b-493d-8740-cefc56decd5d","Type":"ContainerDied","Data":"1a9725d397b304bf017660bada6718b912b9cb3494b0ac8f2147e4750430d9a3"}
Jul 16 22:36:40.325435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:40.325383    2555 scope.go:117] "RemoveContainer" containerID="7e43af8ab21140a8dffa760359b238c8a65bdcd308e7ead2f238c35c0712cfd8"
Jul 16 22:36:40.325718 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:40.325698    2555 scope.go:117] "RemoveContainer" containerID="1a9725d397b304bf017660bada6718b912b9cb3494b0ac8f2147e4750430d9a3"
Jul 16 22:36:40.325970 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:40.325944    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn_odh-ai-gateway-infra(9ed6048f-440b-493d-8740-cefc56decd5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" podUID="9ed6048f-440b-493d-8740-cefc56decd5d"
Jul 16 22:36:41.370505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:41.370474    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:41.508208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:41.508180    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:41.529815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:41.529781    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca\") pod \"9ed6048f-440b-493d-8740-cefc56decd5d\" (UID: \"9ed6048f-440b-493d-8740-cefc56decd5d\") "
Jul 16 22:36:41.530086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:41.530064    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9ed6048f-440b-493d-8740-cefc56decd5d" (UID: "9ed6048f-440b-493d-8740-cefc56decd5d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:41.630755 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:41.630666    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9ed6048f-440b-493d-8740-cefc56decd5d-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:42.338229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:42.338198    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn" event={"ID":"9ed6048f-440b-493d-8740-cefc56decd5d","Type":"ContainerDied","Data":"f818a86e628abe3dcb4b16216b2ae583c51bf3885be26da5641bd2a22a94028f"}
Jul 16 22:36:42.338229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:42.338224    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"
Jul 16 22:36:42.338545 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:42.338258    2555 scope.go:117] "RemoveContainer" containerID="1a9725d397b304bf017660bada6718b912b9cb3494b0ac8f2147e4750430d9a3"
Jul 16 22:36:42.360890 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:42.360864    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:42.372834 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:42.372810    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hmshn"]
Jul 16 22:36:43.676441 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:43.676396    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" path="/var/lib/kubelet/pods/9ed6048f-440b-493d-8740-cefc56decd5d/volumes"
Jul 16 22:36:44.223654 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223620    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"]
Jul 16 22:36:44.223964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223951    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.223964 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223964    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223972    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223979    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223991    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.223997    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224005    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224010    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224016    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224021    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224028    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224033    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224039    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224044    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224050    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d1ea759-df06-4cd5-a183-4e456ddae446" containerName="main"
Jul 16 22:36:44.224056 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224055    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d1ea759-df06-4cd5-a183-4e456ddae446" containerName="main"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224115    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeba430d-a023-451a-bcf7-5ca23e1c9fb8" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224125    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224130    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224136    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224143    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224148    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="272b15c2-b21d-4c7c-b4eb-ae9e5dbeb2c5" containerName="revoke-keys"
Jul 16 22:36:44.224628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.224153    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d1ea759-df06-4cd5-a183-4e456ddae446" containerName="main"
Jul 16 22:36:44.226976 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.226962    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:44.230990 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.230962    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:36:44.231200 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.231183    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:36:44.239023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.238968    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"]
Jul 16 22:36:44.256233 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.256195    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5\" (UID: \"bf964e33-aa00-4916-b5e7-2f093c03165b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:44.357293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.357260    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5\" (UID: \"bf964e33-aa00-4916-b5e7-2f093c03165b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:44.357954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.357936    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5\" (UID: \"bf964e33-aa00-4916-b5e7-2f093c03165b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:44.536885 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.536852    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:44.689657 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:44.689625    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf964e33_aa00_4916_b5e7_2f093c03165b.slice/crio-e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0 WatchSource:0}: Error finding container e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0: Status 404 returned error can't find the container with id e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0
Jul 16 22:36:44.690428 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:44.690404    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"]
Jul 16 22:36:45.357365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:45.357333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5" event={"ID":"bf964e33-aa00-4916-b5e7-2f093c03165b","Type":"ContainerStarted","Data":"e946cf0823d4317e0735ccbb193f930aaa3168c0ff1379ddc54fbf67ec6c4d21"}
Jul 16 22:36:45.357556 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:45.357386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5" event={"ID":"bf964e33-aa00-4916-b5e7-2f093c03165b","Type":"ContainerStarted","Data":"e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0"}
Jul 16 22:36:45.378567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:45.378514    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5" podStartSLOduration=1.3784960179999999 podStartE2EDuration="1.378496018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:36:45.378322346 +0000 UTC m=+2860.215376146" watchObservedRunningTime="2026-07-16 22:36:45.378496018 +0000 UTC m=+2860.215549821"
Jul 16 22:36:46.363589 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:46.363550    2555 generic.go:356] "Generic (PLEG): container finished" podID="bf964e33-aa00-4916-b5e7-2f093c03165b" containerID="e946cf0823d4317e0735ccbb193f930aaa3168c0ff1379ddc54fbf67ec6c4d21" exitCode=0
Jul 16 22:36:46.364052 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:46.363646    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5" event={"ID":"bf964e33-aa00-4916-b5e7-2f093c03165b","Type":"ContainerDied","Data":"e946cf0823d4317e0735ccbb193f930aaa3168c0ff1379ddc54fbf67ec6c4d21"}
Jul 16 22:36:47.498044 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:47.498021    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:47.580728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:47.580691    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca\") pod \"bf964e33-aa00-4916-b5e7-2f093c03165b\" (UID: \"bf964e33-aa00-4916-b5e7-2f093c03165b\") "
Jul 16 22:36:47.581092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:47.581071    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bf964e33-aa00-4916-b5e7-2f093c03165b" (UID: "bf964e33-aa00-4916-b5e7-2f093c03165b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:36:47.681782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:47.681698    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bf964e33-aa00-4916-b5e7-2f093c03165b-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:36:48.376454 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:48.376422    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5"
Jul 16 22:36:48.376454 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:48.376427    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5" event={"ID":"bf964e33-aa00-4916-b5e7-2f093c03165b","Type":"ContainerDied","Data":"e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0"}
Jul 16 22:36:48.376749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:48.376527    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e94b33ba69fcef26684d98de0a64869e7baa88a729fb15a051b611cda82168c0"
Jul 16 22:36:51.178310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.178281    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:36:51.178696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.178652    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf964e33-aa00-4916-b5e7-2f093c03165b" containerName="revoke-keys"
Jul 16 22:36:51.178696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.178664    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf964e33-aa00-4916-b5e7-2f093c03165b" containerName="revoke-keys"
Jul 16 22:36:51.178765 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.178730    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf964e33-aa00-4916-b5e7-2f093c03165b" containerName="revoke-keys"
Jul 16 22:36:51.178765 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.178737    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ed6048f-440b-493d-8740-cefc56decd5d" containerName="revoke-keys"
Jul 16 22:36:51.181877 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.181859    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:36:51.184383 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.184364    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:36:51.184559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.184424    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:36:51.192574 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.192551    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:36:51.211630 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.211603    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-nbl9j\" (UID: \"2b53d3a5-ce20-41c7-af45-001a84ec0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:36:51.312400 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.312365    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-nbl9j\" (UID: \"2b53d3a5-ce20-41c7-af45-001a84ec0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:36:51.312989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.312970    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-nbl9j\" (UID: \"2b53d3a5-ce20-41c7-af45-001a84ec0863\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:36:51.493040 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.492954    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:36:51.630989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:51.630960    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:36:51.632439 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:51.632407    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b53d3a5_ce20_41c7_af45_001a84ec0863.slice/crio-8b4b4ea52346ac4ed1b5826c7a8d60dd2d6fc4dbdc3061466a499cf1daaeda0f WatchSource:0}: Error finding container 8b4b4ea52346ac4ed1b5826c7a8d60dd2d6fc4dbdc3061466a499cf1daaeda0f: Status 404 returned error can't find the container with id 8b4b4ea52346ac4ed1b5826c7a8d60dd2d6fc4dbdc3061466a499cf1daaeda0f
Jul 16 22:36:52.400872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:52.400839    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerID="e7ee60e3d598f1d8b451784e18864df02a94138edf009313a387726d1a2c754f" exitCode=6
Jul 16 22:36:52.401263 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:52.400916    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" event={"ID":"2b53d3a5-ce20-41c7-af45-001a84ec0863","Type":"ContainerDied","Data":"e7ee60e3d598f1d8b451784e18864df02a94138edf009313a387726d1a2c754f"}
Jul 16 22:36:52.401263 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:52.400952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" event={"ID":"2b53d3a5-ce20-41c7-af45-001a84ec0863","Type":"ContainerStarted","Data":"8b4b4ea52346ac4ed1b5826c7a8d60dd2d6fc4dbdc3061466a499cf1daaeda0f"}
Jul 16 22:36:52.401263 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:52.401072    2555 scope.go:117] "RemoveContainer" containerID="e7ee60e3d598f1d8b451784e18864df02a94138edf009313a387726d1a2c754f"
Jul 16 22:36:53.410966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:53.410934    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerID="1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9" exitCode=6
Jul 16 22:36:53.411414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:53.411072    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" event={"ID":"2b53d3a5-ce20-41c7-af45-001a84ec0863","Type":"ContainerDied","Data":"1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9"}
Jul 16 22:36:53.411414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:53.411149    2555 scope.go:117] "RemoveContainer" containerID="e7ee60e3d598f1d8b451784e18864df02a94138edf009313a387726d1a2c754f"
Jul 16 22:36:53.411414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:53.411234    2555 scope.go:117] "RemoveContainer" containerID="1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9"
Jul 16 22:36:53.411414 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:53.411406    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-nbl9j_odh-ai-gateway-infra(2b53d3a5-ce20-41c7-af45-001a84ec0863)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863"
Jul 16 22:36:54.418695 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:54.418664    2555 scope.go:117] "RemoveContainer" containerID="1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9"
Jul 16 22:36:54.419084 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:36:54.418872    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-nbl9j_odh-ai-gateway-infra(2b53d3a5-ce20-41c7-af45-001a84ec0863)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863"
Jul 16 22:36:58.201561 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:58.201455    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:36:59.447911 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.447875    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:36:59.453072 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.453050    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:36:59.462065 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.462041    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:36:59.588350 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.588316    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c\" (UID: \"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:36:59.689021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.688983    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c\" (UID: \"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:36:59.689612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.689593    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c\" (UID: \"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:36:59.764081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.764003    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:36:59.908419 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:36:59.908390    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:36:59.909318 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:36:59.909290    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55ee7c2a_8170_4afc_8258_0a7fb8f48d2f.slice/crio-1a0610e740282484f6baf3e40fc82dd9f8f717e07966a2873c73c6aeaf83e978 WatchSource:0}: Error finding container 1a0610e740282484f6baf3e40fc82dd9f8f717e07966a2873c73c6aeaf83e978: Status 404 returned error can't find the container with id 1a0610e740282484f6baf3e40fc82dd9f8f717e07966a2873c73c6aeaf83e978
Jul 16 22:37:00.452394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:00.452302    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerStarted","Data":"90d4d881107f4d24eb207761e8777436ec01065e6a7d04d399c3d60999c6411d"}
Jul 16 22:37:00.452394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:00.452353    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerStarted","Data":"1a0610e740282484f6baf3e40fc82dd9f8f717e07966a2873c73c6aeaf83e978"}
Jul 16 22:37:00.469925 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:00.469875    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" podStartSLOduration=1.469858725 podStartE2EDuration="1.469858725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:00.467790999 +0000 UTC m=+2875.304844801" watchObservedRunningTime="2026-07-16 22:37:00.469858725 +0000 UTC m=+2875.306912525"
Jul 16 22:37:01.459097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:01.459061    2555 generic.go:356] "Generic (PLEG): container finished" podID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerID="90d4d881107f4d24eb207761e8777436ec01065e6a7d04d399c3d60999c6411d" exitCode=6
Jul 16 22:37:01.459520 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:01.459153    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerDied","Data":"90d4d881107f4d24eb207761e8777436ec01065e6a7d04d399c3d60999c6411d"}
Jul 16 22:37:01.459520 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:01.459309    2555 scope.go:117] "RemoveContainer" containerID="90d4d881107f4d24eb207761e8777436ec01065e6a7d04d399c3d60999c6411d"
Jul 16 22:37:02.466396 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:02.466360    2555 generic.go:356] "Generic (PLEG): container finished" podID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerID="633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0" exitCode=6
Jul 16 22:37:02.466864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:02.466401    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerDied","Data":"633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0"}
Jul 16 22:37:02.466864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:02.466487    2555 scope.go:117] "RemoveContainer" containerID="90d4d881107f4d24eb207761e8777436ec01065e6a7d04d399c3d60999c6411d"
Jul 16 22:37:02.466864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:02.466650    2555 scope.go:117] "RemoveContainer" containerID="633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0"
Jul 16 22:37:02.467091 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:02.466864    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c_odh-ai-gateway-infra(55ee7c2a-8170-4afc-8258-0a7fb8f48d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f"
Jul 16 22:37:03.473708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:03.473678    2555 scope.go:117] "RemoveContainer" containerID="633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0"
Jul 16 22:37:03.474087 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:03.473860    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c_odh-ai-gateway-infra(55ee7c2a-8170-4afc-8258-0a7fb8f48d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f"
Jul 16 22:37:07.676536 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:07.676509    2555 scope.go:117] "RemoveContainer" containerID="1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9"
Jul 16 22:37:08.501399 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:08.501367    2555 generic.go:356] "Generic (PLEG): container finished" podID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerID="b288cf722f6fb2d0264908f515e4e83564248fcaf59d39fb427854e4fcb57f95" exitCode=6
Jul 16 22:37:08.501585 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:08.501446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" event={"ID":"2b53d3a5-ce20-41c7-af45-001a84ec0863","Type":"ContainerDied","Data":"b288cf722f6fb2d0264908f515e4e83564248fcaf59d39fb427854e4fcb57f95"}
Jul 16 22:37:08.501585 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:08.501521    2555 scope.go:117] "RemoveContainer" containerID="1190075df29ed7ffece7030a7bc19346c37237032b0e8507c56b83c402a311f9"
Jul 16 22:37:08.501860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:08.501829    2555 scope.go:117] "RemoveContainer" containerID="b288cf722f6fb2d0264908f515e4e83564248fcaf59d39fb427854e4fcb57f95"
Jul 16 22:37:08.502066 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:08.502041    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-nbl9j_odh-ai-gateway-infra(2b53d3a5-ce20-41c7-af45-001a84ec0863)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863"
Jul 16 22:37:09.537991 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:09.537958    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:37:09.677087 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:09.677053    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:37:09.784114 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:09.784072    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca\") pod \"2b53d3a5-ce20-41c7-af45-001a84ec0863\" (UID: \"2b53d3a5-ce20-41c7-af45-001a84ec0863\") "
Jul 16 22:37:09.784697 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:09.784449    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2b53d3a5-ce20-41c7-af45-001a84ec0863" (UID: "2b53d3a5-ce20-41c7-af45-001a84ec0863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:09.885180 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:09.885104    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2b53d3a5-ce20-41c7-af45-001a84ec0863-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:37:10.515272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:10.515239    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"
Jul 16 22:37:10.515272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:10.515249    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j" event={"ID":"2b53d3a5-ce20-41c7-af45-001a84ec0863","Type":"ContainerDied","Data":"8b4b4ea52346ac4ed1b5826c7a8d60dd2d6fc4dbdc3061466a499cf1daaeda0f"}
Jul 16 22:37:10.515576 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:10.515308    2555 scope.go:117] "RemoveContainer" containerID="b288cf722f6fb2d0264908f515e4e83564248fcaf59d39fb427854e4fcb57f95"
Jul 16 22:37:10.537876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:10.537848    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:37:10.541825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:10.541801    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-nbl9j"]
Jul 16 22:37:11.676305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:11.676263    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" path="/var/lib/kubelet/pods/2b53d3a5-ce20-41c7-af45-001a84ec0863/volumes"
Jul 16 22:37:15.673342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:15.673316    2555 scope.go:117] "RemoveContainer" containerID="633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0"
Jul 16 22:37:16.548197 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:16.548156    2555 generic.go:356] "Generic (PLEG): container finished" podID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerID="6c556fd1213aac9cee724ce7b403a158415e159121090c9521504b0a8f7734e5" exitCode=6
Jul 16 22:37:16.548377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:16.548226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerDied","Data":"6c556fd1213aac9cee724ce7b403a158415e159121090c9521504b0a8f7734e5"}
Jul 16 22:37:16.548377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:16.548291    2555 scope.go:117] "RemoveContainer" containerID="633bad03e50e598ca1d6f4f5f1ae18c9124381a2bf032af50f7994fd847ee6e0"
Jul 16 22:37:16.548677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:16.548659    2555 scope.go:117] "RemoveContainer" containerID="6c556fd1213aac9cee724ce7b403a158415e159121090c9521504b0a8f7734e5"
Jul 16 22:37:16.548868 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:16.548845    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c_odh-ai-gateway-infra(55ee7c2a-8170-4afc-8258-0a7fb8f48d2f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f"
Jul 16 22:37:17.575497 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:17.575450    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:37:17.711708 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:17.711686    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:37:17.862409 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:17.862316    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca\") pod \"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f\" (UID: \"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f\") "
Jul 16 22:37:17.862725 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:17.862700    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" (UID: "55ee7c2a-8170-4afc-8258-0a7fb8f48d2f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:17.963872 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:17.963835    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:37:18.563658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.563628    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"
Jul 16 22:37:18.563855 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.563669    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c" event={"ID":"55ee7c2a-8170-4afc-8258-0a7fb8f48d2f","Type":"ContainerDied","Data":"1a0610e740282484f6baf3e40fc82dd9f8f717e07966a2873c73c6aeaf83e978"}
Jul 16 22:37:18.563855 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.563731    2555 scope.go:117] "RemoveContainer" containerID="6c556fd1213aac9cee724ce7b403a158415e159121090c9521504b0a8f7734e5"
Jul 16 22:37:18.585058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.585030    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:37:18.592639 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.592613    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mnh4c"]
Jul 16 22:37:18.912410 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:18.912327    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:37:19.676138 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:19.676109    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" path="/var/lib/kubelet/pods/55ee7c2a-8170-4afc-8258-0a7fb8f48d2f/volumes"
Jul 16 22:37:20.639311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639278    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:20.639790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639769    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639789    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639802    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639807    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639813    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639834    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639848    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639910    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639920    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639925    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639931    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639937    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:20.639989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.639941    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55ee7c2a-8170-4afc-8258-0a7fb8f48d2f" containerName="revoke-keys"
Jul 16 22:37:20.644288 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.644270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:20.646831 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.646808    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:37:20.646995 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.646970    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:37:20.651217 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.651187    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:20.792419 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.792384    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm\" (UID: \"b0f28862-1dd5-4e4a-8b5b-102961bf0b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:20.893886 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.893800    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm\" (UID: \"b0f28862-1dd5-4e4a-8b5b-102961bf0b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:20.894357 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.894340    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm\" (UID: \"b0f28862-1dd5-4e4a-8b5b-102961bf0b65\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:20.957185 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:20.957156    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:21.100346 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:21.100318    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:21.102130 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:37:21.102089    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0f28862_1dd5_4e4a_8b5b_102961bf0b65.slice/crio-5788a3aa01fa038162e2db4966ce77d973a979c01e7aa203d4545adc07ee567c WatchSource:0}: Error finding container 5788a3aa01fa038162e2db4966ce77d973a979c01e7aa203d4545adc07ee567c: Status 404 returned error can't find the container with id 5788a3aa01fa038162e2db4966ce77d973a979c01e7aa203d4545adc07ee567c
Jul 16 22:37:21.583104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:21.583007    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerStarted","Data":"c481cb8eb5245aa7da0c49e85f27653da72c26982624f733c8851968f3cede38"}
Jul 16 22:37:21.583104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:21.583076    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerStarted","Data":"5788a3aa01fa038162e2db4966ce77d973a979c01e7aa203d4545adc07ee567c"}
Jul 16 22:37:21.599587 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:21.599542    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" podStartSLOduration=1.599528286 podStartE2EDuration="1.599528286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:37:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:37:21.598415712 +0000 UTC m=+2896.435469515" watchObservedRunningTime="2026-07-16 22:37:21.599528286 +0000 UTC m=+2896.436582086"
Jul 16 22:37:22.589992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:22.589949    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerID="c481cb8eb5245aa7da0c49e85f27653da72c26982624f733c8851968f3cede38" exitCode=6
Jul 16 22:37:22.590557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:22.590000    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerDied","Data":"c481cb8eb5245aa7da0c49e85f27653da72c26982624f733c8851968f3cede38"}
Jul 16 22:37:22.590557 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:22.590281    2555 scope.go:117] "RemoveContainer" containerID="c481cb8eb5245aa7da0c49e85f27653da72c26982624f733c8851968f3cede38"
Jul 16 22:37:23.597342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:23.597309    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerID="4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3" exitCode=6
Jul 16 22:37:23.597819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:23.597376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerDied","Data":"4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3"}
Jul 16 22:37:23.597819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:23.597438    2555 scope.go:117] "RemoveContainer" containerID="c481cb8eb5245aa7da0c49e85f27653da72c26982624f733c8851968f3cede38"
Jul 16 22:37:23.597819 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:23.597652    2555 scope.go:117] "RemoveContainer" containerID="4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3"
Jul 16 22:37:23.597943 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:23.597875    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm_odh-ai-gateway-infra(b0f28862-1dd5-4e4a-8b5b-102961bf0b65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65"
Jul 16 22:37:24.604093 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:24.604064    2555 scope.go:117] "RemoveContainer" containerID="4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3"
Jul 16 22:37:24.604585 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:24.604261    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm_odh-ai-gateway-infra(b0f28862-1dd5-4e4a-8b5b-102961bf0b65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65"
Jul 16 22:37:29.324933 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:29.324889    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:37:34.474970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.474928    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"]
Jul 16 22:37:34.475382 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.475342    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:34.475382 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.475356    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2b53d3a5-ce20-41c7-af45-001a84ec0863" containerName="revoke-keys"
Jul 16 22:37:34.479787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.479772    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:34.486390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.486368    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"]
Jul 16 22:37:34.512440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.512412    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27\" (UID: \"ead0518e-a662-4ad4-b5d5-d50aad9da903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:34.613559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.613531    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27\" (UID: \"ead0518e-a662-4ad4-b5d5-d50aad9da903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:34.614114 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.614096    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27\" (UID: \"ead0518e-a662-4ad4-b5d5-d50aad9da903\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:34.790709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.790681    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:34.939203 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:34.939170    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"]
Jul 16 22:37:34.939602 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:37:34.939577    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podead0518e_a662_4ad4_b5d5_d50aad9da903.slice/crio-ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec WatchSource:0}: Error finding container ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec: Status 404 returned error can't find the container with id ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec
Jul 16 22:37:35.662772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:35.662733    2555 generic.go:356] "Generic (PLEG): container finished" podID="ead0518e-a662-4ad4-b5d5-d50aad9da903" containerID="b3a78495166f6f1bb074a6556ae170367e8e625eb0022a2f5eb3560175de3557" exitCode=0
Jul 16 22:37:35.663155 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:35.662784    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27" event={"ID":"ead0518e-a662-4ad4-b5d5-d50aad9da903","Type":"ContainerDied","Data":"b3a78495166f6f1bb074a6556ae170367e8e625eb0022a2f5eb3560175de3557"}
Jul 16 22:37:35.663155 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:35.662851    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27" event={"ID":"ead0518e-a662-4ad4-b5d5-d50aad9da903","Type":"ContainerStarted","Data":"ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec"}
Jul 16 22:37:36.802260 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:36.802231    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:36.833386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:36.833354    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca\") pod \"ead0518e-a662-4ad4-b5d5-d50aad9da903\" (UID: \"ead0518e-a662-4ad4-b5d5-d50aad9da903\") "
Jul 16 22:37:36.833798 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:36.833772    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ead0518e-a662-4ad4-b5d5-d50aad9da903" (UID: "ead0518e-a662-4ad4-b5d5-d50aad9da903"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:36.934644 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:36.934560    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ead0518e-a662-4ad4-b5d5-d50aad9da903-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:37:37.675774 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:37.675746    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27"
Jul 16 22:37:37.676106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:37.676081    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27" event={"ID":"ead0518e-a662-4ad4-b5d5-d50aad9da903","Type":"ContainerDied","Data":"ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec"}
Jul 16 22:37:37.676154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:37.676131    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed389bc8c8b3ada7fec6d1c164fa739730ef1d178ecd42145d8b37669612d7ec"
Jul 16 22:37:39.670353 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:39.670321    2555 scope.go:117] "RemoveContainer" containerID="4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3"
Jul 16 22:37:40.695532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:40.695439    2555 generic.go:356] "Generic (PLEG): container finished" podID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerID="cc63a4fa2c4afdc677d915f9c1f0b4531bb725ac4ca4d6f7191b4b29717ed304" exitCode=6
Jul 16 22:37:40.695532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:40.695512    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerDied","Data":"cc63a4fa2c4afdc677d915f9c1f0b4531bb725ac4ca4d6f7191b4b29717ed304"}
Jul 16 22:37:40.696039 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:40.695573    2555 scope.go:117] "RemoveContainer" containerID="4c55ca0c4d3fa9e734142ae74423e52a0f47f2c933e8a7bd06985599165f30f3"
Jul 16 22:37:40.696039 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:40.695884    2555 scope.go:117] "RemoveContainer" containerID="cc63a4fa2c4afdc677d915f9c1f0b4531bb725ac4ca4d6f7191b4b29717ed304"
Jul 16 22:37:40.696148 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:40.696120    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm_odh-ai-gateway-infra(b0f28862-1dd5-4e4a-8b5b-102961bf0b65)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65"
Jul 16 22:37:41.223714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.223676    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:37:41.224039 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.224027    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ead0518e-a662-4ad4-b5d5-d50aad9da903" containerName="revoke-keys"
Jul 16 22:37:41.224080 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.224041    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ead0518e-a662-4ad4-b5d5-d50aad9da903" containerName="revoke-keys"
Jul 16 22:37:41.224135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.224126    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ead0518e-a662-4ad4-b5d5-d50aad9da903" containerName="revoke-keys"
Jul 16 22:37:41.228427 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.228408    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:41.239511 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.239480    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:37:41.273538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.273508    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-542nq\" (UID: \"12301187-6e46-4bc2-ae7b-9818487289eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:41.374642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.374607    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-542nq\" (UID: \"12301187-6e46-4bc2-ae7b-9818487289eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:41.375248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.375227    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-542nq\" (UID: \"12301187-6e46-4bc2-ae7b-9818487289eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:41.540589 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.540550    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:41.679187 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.679165    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:37:41.681204 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:37:41.681177    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12301187_6e46_4bc2_ae7b_9818487289eb.slice/crio-98d3578b3c16ecd53947a7b0ac2097a8be3385330ca1018ff4b73a1296026dfd WatchSource:0}: Error finding container 98d3578b3c16ecd53947a7b0ac2097a8be3385330ca1018ff4b73a1296026dfd: Status 404 returned error can't find the container with id 98d3578b3c16ecd53947a7b0ac2097a8be3385330ca1018ff4b73a1296026dfd
Jul 16 22:37:41.705761 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.705001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" event={"ID":"12301187-6e46-4bc2-ae7b-9818487289eb","Type":"ContainerStarted","Data":"98d3578b3c16ecd53947a7b0ac2097a8be3385330ca1018ff4b73a1296026dfd"}
Jul 16 22:37:41.726184 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.726159    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:41.858368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.858346    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:41.981377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.981340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca\") pod \"b0f28862-1dd5-4e4a-8b5b-102961bf0b65\" (UID: \"b0f28862-1dd5-4e4a-8b5b-102961bf0b65\") "
Jul 16 22:37:41.981725 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:41.981703    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b0f28862-1dd5-4e4a-8b5b-102961bf0b65" (UID: "b0f28862-1dd5-4e4a-8b5b-102961bf0b65"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:42.082014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.081982    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b0f28862-1dd5-4e4a-8b5b-102961bf0b65-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:37:42.714958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.714868    2555 generic.go:356] "Generic (PLEG): container finished" podID="12301187-6e46-4bc2-ae7b-9818487289eb" containerID="06c15df15b9b65e9066ba60fc14ec5d35d055d89f0ceb30e037da0ab72871630" exitCode=6
Jul 16 22:37:42.715365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.714951    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" event={"ID":"12301187-6e46-4bc2-ae7b-9818487289eb","Type":"ContainerDied","Data":"06c15df15b9b65e9066ba60fc14ec5d35d055d89f0ceb30e037da0ab72871630"}
Jul 16 22:37:42.715365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.715158    2555 scope.go:117] "RemoveContainer" containerID="06c15df15b9b65e9066ba60fc14ec5d35d055d89f0ceb30e037da0ab72871630"
Jul 16 22:37:42.716317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.716299    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"
Jul 16 22:37:42.716408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.716343    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm" event={"ID":"b0f28862-1dd5-4e4a-8b5b-102961bf0b65","Type":"ContainerDied","Data":"5788a3aa01fa038162e2db4966ce77d973a979c01e7aa203d4545adc07ee567c"}
Jul 16 22:37:42.716408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.716390    2555 scope.go:117] "RemoveContainer" containerID="cc63a4fa2c4afdc677d915f9c1f0b4531bb725ac4ca4d6f7191b4b29717ed304"
Jul 16 22:37:42.765552 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.765508    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:42.767139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:42.767118    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-86nkm"]
Jul 16 22:37:43.677102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:43.677019    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" path="/var/lib/kubelet/pods/b0f28862-1dd5-4e4a-8b5b-102961bf0b65/volumes"
Jul 16 22:37:43.724394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:43.724362    2555 generic.go:356] "Generic (PLEG): container finished" podID="12301187-6e46-4bc2-ae7b-9818487289eb" containerID="e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b" exitCode=6
Jul 16 22:37:43.724891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:43.724457    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" event={"ID":"12301187-6e46-4bc2-ae7b-9818487289eb","Type":"ContainerDied","Data":"e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b"}
Jul 16 22:37:43.724891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:43.724545    2555 scope.go:117] "RemoveContainer" containerID="06c15df15b9b65e9066ba60fc14ec5d35d055d89f0ceb30e037da0ab72871630"
Jul 16 22:37:43.724891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:43.724700    2555 scope.go:117] "RemoveContainer" containerID="e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b"
Jul 16 22:37:43.725044 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:43.724969    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-542nq_odh-ai-gateway-infra(12301187-6e46-4bc2-ae7b-9818487289eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" podUID="12301187-6e46-4bc2-ae7b-9818487289eb"
Jul 16 22:37:44.733185 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:44.733158    2555 scope.go:117] "RemoveContainer" containerID="e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b"
Jul 16 22:37:44.733713 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:44.733410    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-542nq_odh-ai-gateway-infra(12301187-6e46-4bc2-ae7b-9818487289eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" podUID="12301187-6e46-4bc2-ae7b-9818487289eb"
Jul 16 22:37:48.795379 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:48.795338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:37:49.455262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455225    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:37:49.455612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455598    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455614    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455632    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455638    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455655    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455662    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455722    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.455861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.455730    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.459999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.459972    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:37:49.475868 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.475841    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:37:49.504642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.504602    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:37:49.505092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.505074    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b0f28862-1dd5-4e4a-8b5b-102961bf0b65" containerName="revoke-keys"
Jul 16 22:37:49.508042 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.508020    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:37:49.527116 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.527089    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:37:49.549202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.549173    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn\" (UID: \"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:37:49.549388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.549213    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx\" (UID: \"16e041cc-c1c4-46f1-8de2-e21994a2adea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:37:49.650645 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.650595    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn\" (UID: \"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:37:49.650645 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.650654    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx\" (UID: \"16e041cc-c1c4-46f1-8de2-e21994a2adea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:37:49.651251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.651232    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn\" (UID: \"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:37:49.651292 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.651243    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx\" (UID: \"16e041cc-c1c4-46f1-8de2-e21994a2adea\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:37:49.770029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.769947    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:37:49.818045 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.818017    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:37:49.965735 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:49.965708    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:37:49.967618 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:37:49.967589    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16e041cc_c1c4_46f1_8de2_e21994a2adea.slice/crio-dfe399251c027f26a865c334f66b3de56c85578d50054bc4b25969235d35f8ea WatchSource:0}: Error finding container dfe399251c027f26a865c334f66b3de56c85578d50054bc4b25969235d35f8ea: Status 404 returned error can't find the container with id dfe399251c027f26a865c334f66b3de56c85578d50054bc4b25969235d35f8ea
Jul 16 22:37:50.129904 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.129875    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:37:50.131376 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:37:50.131351    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d8d8e0d_ccdc_4dbb_8d56_e4f133045d89.slice/crio-08b57b76315667f9a4f999a07708deff0da6c5e30ccfe6c2dab035cd45186694 WatchSource:0}: Error finding container 08b57b76315667f9a4f999a07708deff0da6c5e30ccfe6c2dab035cd45186694: Status 404 returned error can't find the container with id 08b57b76315667f9a4f999a07708deff0da6c5e30ccfe6c2dab035cd45186694
Jul 16 22:37:50.768400 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.768363    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerID="818bfbbab5f43c1e6a6b752fe3b0df2a5840afdd9166e2ee7ad64521bcea9032" exitCode=6
Jul 16 22:37:50.768603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.768403    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" event={"ID":"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89","Type":"ContainerDied","Data":"818bfbbab5f43c1e6a6b752fe3b0df2a5840afdd9166e2ee7ad64521bcea9032"}
Jul 16 22:37:50.768603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.768500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" event={"ID":"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89","Type":"ContainerStarted","Data":"08b57b76315667f9a4f999a07708deff0da6c5e30ccfe6c2dab035cd45186694"}
Jul 16 22:37:50.768770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.768755    2555 scope.go:117] "RemoveContainer" containerID="818bfbbab5f43c1e6a6b752fe3b0df2a5840afdd9166e2ee7ad64521bcea9032"
Jul 16 22:37:50.770133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.770110    2555 generic.go:356] "Generic (PLEG): container finished" podID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerID="d9057664165f9ae669dbcd65ba3cffc59ead9eca27e1368deaf9eef5cd7a7a65" exitCode=6
Jul 16 22:37:50.770234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.770178    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" event={"ID":"16e041cc-c1c4-46f1-8de2-e21994a2adea","Type":"ContainerDied","Data":"d9057664165f9ae669dbcd65ba3cffc59ead9eca27e1368deaf9eef5cd7a7a65"}
Jul 16 22:37:50.770234 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.770225    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" event={"ID":"16e041cc-c1c4-46f1-8de2-e21994a2adea","Type":"ContainerStarted","Data":"dfe399251c027f26a865c334f66b3de56c85578d50054bc4b25969235d35f8ea"}
Jul 16 22:37:50.770381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:50.770368    2555 scope.go:117] "RemoveContainer" containerID="d9057664165f9ae669dbcd65ba3cffc59ead9eca27e1368deaf9eef5cd7a7a65"
Jul 16 22:37:51.777210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.777183    2555 generic.go:356] "Generic (PLEG): container finished" podID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerID="f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f" exitCode=6
Jul 16 22:37:51.777619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.777260    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" event={"ID":"16e041cc-c1c4-46f1-8de2-e21994a2adea","Type":"ContainerDied","Data":"f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f"}
Jul 16 22:37:51.777619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.777323    2555 scope.go:117] "RemoveContainer" containerID="d9057664165f9ae669dbcd65ba3cffc59ead9eca27e1368deaf9eef5cd7a7a65"
Jul 16 22:37:51.777619 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.777553    2555 scope.go:117] "RemoveContainer" containerID="f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f"
Jul 16 22:37:51.777778 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:51.777757    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx_odh-ai-gateway-infra(16e041cc-c1c4-46f1-8de2-e21994a2adea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea"
Jul 16 22:37:51.779310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.779287    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerID="97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706" exitCode=6
Jul 16 22:37:51.779409 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.779333    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" event={"ID":"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89","Type":"ContainerDied","Data":"97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706"}
Jul 16 22:37:51.779551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.779538    2555 scope.go:117] "RemoveContainer" containerID="97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706"
Jul 16 22:37:51.779726 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:51.779710    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn_odh-ai-gateway-infra(1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89"
Jul 16 22:37:51.789099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:51.789080    2555 scope.go:117] "RemoveContainer" containerID="818bfbbab5f43c1e6a6b752fe3b0df2a5840afdd9166e2ee7ad64521bcea9032"
Jul 16 22:37:52.786229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:52.786190    2555 scope.go:117] "RemoveContainer" containerID="97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706"
Jul 16 22:37:52.786642 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:52.786421    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn_odh-ai-gateway-infra(1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89"
Jul 16 22:37:52.787698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:52.787681    2555 scope.go:117] "RemoveContainer" containerID="f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f"
Jul 16 22:37:52.787840 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:52.787823    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx_odh-ai-gateway-infra(16e041cc-c1c4-46f1-8de2-e21994a2adea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea"
Jul 16 22:37:56.670717 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:56.670684    2555 scope.go:117] "RemoveContainer" containerID="e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b"
Jul 16 22:37:57.816963 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:57.816934    2555 generic.go:356] "Generic (PLEG): container finished" podID="12301187-6e46-4bc2-ae7b-9818487289eb" containerID="f8bd463315f592dd4d4df06ca568cc34f6e92f5c36f5dc0aa29a2488434b52e1" exitCode=6
Jul 16 22:37:57.817404 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:57.816991    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" event={"ID":"12301187-6e46-4bc2-ae7b-9818487289eb","Type":"ContainerDied","Data":"f8bd463315f592dd4d4df06ca568cc34f6e92f5c36f5dc0aa29a2488434b52e1"}
Jul 16 22:37:57.817404 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:57.817036    2555 scope.go:117] "RemoveContainer" containerID="e7c3888cd2df24eadc7021d9b2e8ae4e3578e52843f98a1e8602859e0c026e0b"
Jul 16 22:37:57.817404 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:57.817380    2555 scope.go:117] "RemoveContainer" containerID="f8bd463315f592dd4d4df06ca568cc34f6e92f5c36f5dc0aa29a2488434b52e1"
Jul 16 22:37:57.817668 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:37:57.817644    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-542nq_odh-ai-gateway-infra(12301187-6e46-4bc2-ae7b-9818487289eb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" podUID="12301187-6e46-4bc2-ae7b-9818487289eb"
Jul 16 22:37:58.850905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:58.850871    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:37:58.987078 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:58.987055    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:59.137253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.137160    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca\") pod \"12301187-6e46-4bc2-ae7b-9818487289eb\" (UID: \"12301187-6e46-4bc2-ae7b-9818487289eb\") "
Jul 16 22:37:59.137614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.137583    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12301187-6e46-4bc2-ae7b-9818487289eb" (UID: "12301187-6e46-4bc2-ae7b-9818487289eb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:37:59.238711 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.238669    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12301187-6e46-4bc2-ae7b-9818487289eb-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:37:59.833378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.833354    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"
Jul 16 22:37:59.833601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.833347    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq" event={"ID":"12301187-6e46-4bc2-ae7b-9818487289eb","Type":"ContainerDied","Data":"98d3578b3c16ecd53947a7b0ac2097a8be3385330ca1018ff4b73a1296026dfd"}
Jul 16 22:37:59.833601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.833500    2555 scope.go:117] "RemoveContainer" containerID="f8bd463315f592dd4d4df06ca568cc34f6e92f5c36f5dc0aa29a2488434b52e1"
Jul 16 22:37:59.852656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.852633    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:37:59.860962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:37:59.860934    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-542nq"]
Jul 16 22:38:00.778852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.778819    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"]
Jul 16 22:38:00.779214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779199    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779219    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779237    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779242    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779248    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779254    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779350    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.779417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.779358    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:00.784067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.784051    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:00.795736 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.795711    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"]
Jul 16 22:38:00.853318 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.853284    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7\" (UID: \"2976694d-2c26-470d-8447-51eada901341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:00.953959 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.953926    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7\" (UID: \"2976694d-2c26-470d-8447-51eada901341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:00.954610 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:00.954581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7\" (UID: \"2976694d-2c26-470d-8447-51eada901341\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:01.094810 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.094700    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:01.231080 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.231057    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"]
Jul 16 22:38:01.232825 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:38:01.232797    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2976694d_2c26_470d_8447_51eada901341.slice/crio-03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02 WatchSource:0}: Error finding container 03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02: Status 404 returned error can't find the container with id 03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02
Jul 16 22:38:01.677398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.677320    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" path="/var/lib/kubelet/pods/12301187-6e46-4bc2-ae7b-9818487289eb/volumes"
Jul 16 22:38:01.846976 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.846932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7" event={"ID":"2976694d-2c26-470d-8447-51eada901341","Type":"ContainerStarted","Data":"fd35f442295a4d5d51bdde61b75a4968de4bb589006325176dfad21309b41cf4"}
Jul 16 22:38:01.847207 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.847004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7" event={"ID":"2976694d-2c26-470d-8447-51eada901341","Type":"ContainerStarted","Data":"03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02"}
Jul 16 22:38:01.867992 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:01.867938    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7" podStartSLOduration=1.867922413 podStartE2EDuration="1.867922413s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:01.865296577 +0000 UTC m=+2936.702350376" watchObservedRunningTime="2026-07-16 22:38:01.867922413 +0000 UTC m=+2936.704976235"
Jul 16 22:38:02.854583 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:02.854551    2555 generic.go:356] "Generic (PLEG): container finished" podID="2976694d-2c26-470d-8447-51eada901341" containerID="fd35f442295a4d5d51bdde61b75a4968de4bb589006325176dfad21309b41cf4" exitCode=0
Jul 16 22:38:02.854583 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:02.854585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7" event={"ID":"2976694d-2c26-470d-8447-51eada901341","Type":"ContainerDied","Data":"fd35f442295a4d5d51bdde61b75a4968de4bb589006325176dfad21309b41cf4"}
Jul 16 22:38:03.670942 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:03.670910    2555 scope.go:117] "RemoveContainer" containerID="97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706"
Jul 16 22:38:04.002173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.002145    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:04.083033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.083005    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca\") pod \"2976694d-2c26-470d-8447-51eada901341\" (UID: \"2976694d-2c26-470d-8447-51eada901341\") "
Jul 16 22:38:04.083381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.083361    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2976694d-2c26-470d-8447-51eada901341" (UID: "2976694d-2c26-470d-8447-51eada901341"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:04.184605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.184572    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2976694d-2c26-470d-8447-51eada901341-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:38:04.866162 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.866135    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7"
Jul 16 22:38:04.866162 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.866142    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7" event={"ID":"2976694d-2c26-470d-8447-51eada901341","Type":"ContainerDied","Data":"03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02"}
Jul 16 22:38:04.866665 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.866189    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03cb044b2e47e1644019a0073da90809963d11cc6d2ec947a78e1a67b56d9c02"
Jul 16 22:38:04.868181 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.868144    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerID="cf33a6588940ac88810eea9259d66ed922f9fb4088b6b060bfcddce88ab066df" exitCode=6
Jul 16 22:38:04.868298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.868208    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" event={"ID":"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89","Type":"ContainerDied","Data":"cf33a6588940ac88810eea9259d66ed922f9fb4088b6b060bfcddce88ab066df"}
Jul 16 22:38:04.868298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.868255    2555 scope.go:117] "RemoveContainer" containerID="97d231bea096ad78d672bdcec58660902c853940f2592d1071379507e216f706"
Jul 16 22:38:04.868575 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:04.868555    2555 scope.go:117] "RemoveContainer" containerID="cf33a6588940ac88810eea9259d66ed922f9fb4088b6b060bfcddce88ab066df"
Jul 16 22:38:04.868784 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:04.868764    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn_odh-ai-gateway-infra(1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89"
Jul 16 22:38:05.902711 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:05.902682    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:38:06.063118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.063091    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:38:06.202636 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.202615    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca\") pod \"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89\" (UID: \"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89\") "
Jul 16 22:38:06.202936 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.202915    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" (UID: "1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:06.303914 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.303881    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:38:06.670273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.670239    2555 scope.go:117] "RemoveContainer" containerID="f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f"
Jul 16 22:38:06.880918 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.880887    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"
Jul 16 22:38:06.881089 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.880894    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn" event={"ID":"1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89","Type":"ContainerDied","Data":"08b57b76315667f9a4f999a07708deff0da6c5e30ccfe6c2dab035cd45186694"}
Jul 16 22:38:06.881089 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.881017    2555 scope.go:117] "RemoveContainer" containerID="cf33a6588940ac88810eea9259d66ed922f9fb4088b6b060bfcddce88ab066df"
Jul 16 22:38:06.914564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.914526    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:38:06.932194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:06.932111    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jtsgn"]
Jul 16 22:38:07.676137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:07.676056    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" path="/var/lib/kubelet/pods/1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89/volumes"
Jul 16 22:38:07.888660 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:07.888631    2555 generic.go:356] "Generic (PLEG): container finished" podID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerID="f81c91816be7d0254f9b2a9bf32979708183b1d08f03cff7e417f05b3be8a68b" exitCode=6
Jul 16 22:38:07.888802 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:07.888706    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" event={"ID":"16e041cc-c1c4-46f1-8de2-e21994a2adea","Type":"ContainerDied","Data":"f81c91816be7d0254f9b2a9bf32979708183b1d08f03cff7e417f05b3be8a68b"}
Jul 16 22:38:07.888802 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:07.888764    2555 scope.go:117] "RemoveContainer" containerID="f5bfdcf84ef9218039531ecbe8c1a5becca23927cf5497c5f4309e4ef5d2705f"
Jul 16 22:38:07.889075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:07.889060    2555 scope.go:117] "RemoveContainer" containerID="f81c91816be7d0254f9b2a9bf32979708183b1d08f03cff7e417f05b3be8a68b"
Jul 16 22:38:07.889307 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:07.889287    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx_odh-ai-gateway-infra(16e041cc-c1c4-46f1-8de2-e21994a2adea)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea"
Jul 16 22:38:08.930204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:08.930171    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:38:09.065287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.065263    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:38:09.231085 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.230995    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca\") pod \"16e041cc-c1c4-46f1-8de2-e21994a2adea\" (UID: \"16e041cc-c1c4-46f1-8de2-e21994a2adea\") "
Jul 16 22:38:09.231361 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.231337    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "16e041cc-c1c4-46f1-8de2-e21994a2adea" (UID: "16e041cc-c1c4-46f1-8de2-e21994a2adea"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:09.331762 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.331733    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/16e041cc-c1c4-46f1-8de2-e21994a2adea-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:38:09.903724 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.903692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx" event={"ID":"16e041cc-c1c4-46f1-8de2-e21994a2adea","Type":"ContainerDied","Data":"dfe399251c027f26a865c334f66b3de56c85578d50054bc4b25969235d35f8ea"}
Jul 16 22:38:09.903919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.903748    2555 scope.go:117] "RemoveContainer" containerID="f81c91816be7d0254f9b2a9bf32979708183b1d08f03cff7e417f05b3be8a68b"
Jul 16 22:38:09.903919 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.903705    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"
Jul 16 22:38:09.922621 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.922599    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:38:09.929074 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:09.929053    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-qznbx"]
Jul 16 22:38:11.676317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:11.676276    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" path="/var/lib/kubelet/pods/16e041cc-c1c4-46f1-8de2-e21994a2adea/volumes"
Jul 16 22:38:14.284200 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:14.284165    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:38:20.703499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703437    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703818    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703829    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703836    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2976694d-2c26-470d-8447-51eada901341" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703841    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2976694d-2c26-470d-8447-51eada901341" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703849    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703854    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703861    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703867    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703872    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703876    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703882    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703886    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703897    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.703920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703902    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703956    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703963    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703969    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2976694d-2c26-470d-8447-51eada901341" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703974    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703979    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703983    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12301187-6e46-4bc2-ae7b-9818487289eb" containerName="revoke-keys"
Jul 16 22:38:20.704368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.703988    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d8d8e0d-ccdc-4dbb-8d56-e4f133045d89" containerName="revoke-keys"
Jul 16 22:38:20.707105 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.707088    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:20.709706 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.709676    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:38:20.709865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.709708    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:38:20.716317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.716288    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:20.734707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.734654    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj\" (UID: \"42e0a2d9-28c0-428c-ad48-e411bc630bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:20.835790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.835745    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj\" (UID: \"42e0a2d9-28c0-428c-ad48-e411bc630bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:20.836388 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:20.836368    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj\" (UID: \"42e0a2d9-28c0-428c-ad48-e411bc630bb8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:21.019864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.019769    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:21.154367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.154341    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:21.157077 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:38:21.157051    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42e0a2d9_28c0_428c_ad48_e411bc630bb8.slice/crio-0bdcfdbaecb52dec4d61f5c00217805264991677bf2192325e60e5ec38b7b549 WatchSource:0}: Error finding container 0bdcfdbaecb52dec4d61f5c00217805264991677bf2192325e60e5ec38b7b549: Status 404 returned error can't find the container with id 0bdcfdbaecb52dec4d61f5c00217805264991677bf2192325e60e5ec38b7b549
Jul 16 22:38:21.969023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.968931    2555 generic.go:356] "Generic (PLEG): container finished" podID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerID="3adfe6496a0e6d202e4164dfd7bb544f043cac424cad090470b1005738d8829e" exitCode=6
Jul 16 22:38:21.969416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.969017    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerDied","Data":"3adfe6496a0e6d202e4164dfd7bb544f043cac424cad090470b1005738d8829e"}
Jul 16 22:38:21.969416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.969067    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerStarted","Data":"0bdcfdbaecb52dec4d61f5c00217805264991677bf2192325e60e5ec38b7b549"}
Jul 16 22:38:21.969416 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:21.969189    2555 scope.go:117] "RemoveContainer" containerID="3adfe6496a0e6d202e4164dfd7bb544f043cac424cad090470b1005738d8829e"
Jul 16 22:38:22.976608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:22.976508    2555 generic.go:356] "Generic (PLEG): container finished" podID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerID="25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3" exitCode=6
Jul 16 22:38:22.976608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:22.976585    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerDied","Data":"25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3"}
Jul 16 22:38:22.977085 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:22.976642    2555 scope.go:117] "RemoveContainer" containerID="3adfe6496a0e6d202e4164dfd7bb544f043cac424cad090470b1005738d8829e"
Jul 16 22:38:22.977085 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:22.976806    2555 scope.go:117] "RemoveContainer" containerID="25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3"
Jul 16 22:38:22.977085 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:22.977029    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj_odh-ai-gateway-infra(42e0a2d9-28c0-428c-ad48-e411bc630bb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8"
Jul 16 22:38:23.983921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:23.983893    2555 scope.go:117] "RemoveContainer" containerID="25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3"
Jul 16 22:38:23.984325 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:23.984095    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj_odh-ai-gateway-infra(42e0a2d9-28c0-428c-ad48-e411bc630bb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8"
Jul 16 22:38:31.280799 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.280714    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:31.281181 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.281125    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="16e041cc-c1c4-46f1-8de2-e21994a2adea" containerName="revoke-keys"
Jul 16 22:38:31.285628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.285608    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:31.298571 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.298545    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:31.319896 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.319870    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs8ht\" (UID: \"43ff5046-263e-401d-8d6d-03993600d26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:31.420902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.420864    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs8ht\" (UID: \"43ff5046-263e-401d-8d6d-03993600d26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:31.421530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.421507    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-zs8ht\" (UID: \"43ff5046-263e-401d-8d6d-03993600d26a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:31.595658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.595567    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:31.733424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:31.733398    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:31.734190 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:38:31.734165    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43ff5046_263e_401d_8d6d_03993600d26a.slice/crio-008d21b79e37355e27246fc9e70734019a44ccd4d15f28b83c1e68d524f5618c WatchSource:0}: Error finding container 008d21b79e37355e27246fc9e70734019a44ccd4d15f28b83c1e68d524f5618c: Status 404 returned error can't find the container with id 008d21b79e37355e27246fc9e70734019a44ccd4d15f28b83c1e68d524f5618c
Jul 16 22:38:32.031733 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:32.031696    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerStarted","Data":"586309a0891475f7b7232468fc533a1378d42f745adc7917fd6ee7bdcc057d41"}
Jul 16 22:38:32.031921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:32.031785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerStarted","Data":"008d21b79e37355e27246fc9e70734019a44ccd4d15f28b83c1e68d524f5618c"}
Jul 16 22:38:32.053564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:32.053510    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" podStartSLOduration=1.053494955 podStartE2EDuration="1.053494955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:32.052679983 +0000 UTC m=+2966.889733794" watchObservedRunningTime="2026-07-16 22:38:32.053494955 +0000 UTC m=+2966.890548752"
Jul 16 22:38:33.038890 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:33.038855    2555 generic.go:356] "Generic (PLEG): container finished" podID="43ff5046-263e-401d-8d6d-03993600d26a" containerID="586309a0891475f7b7232468fc533a1378d42f745adc7917fd6ee7bdcc057d41" exitCode=6
Jul 16 22:38:33.039300 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:33.038952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerDied","Data":"586309a0891475f7b7232468fc533a1378d42f745adc7917fd6ee7bdcc057d41"}
Jul 16 22:38:33.039300 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:33.039143    2555 scope.go:117] "RemoveContainer" containerID="586309a0891475f7b7232468fc533a1378d42f745adc7917fd6ee7bdcc057d41"
Jul 16 22:38:34.047159 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:34.047126    2555 generic.go:356] "Generic (PLEG): container finished" podID="43ff5046-263e-401d-8d6d-03993600d26a" containerID="081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7" exitCode=6
Jul 16 22:38:34.047538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:34.047206    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerDied","Data":"081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7"}
Jul 16 22:38:34.047538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:34.047262    2555 scope.go:117] "RemoveContainer" containerID="586309a0891475f7b7232468fc533a1378d42f745adc7917fd6ee7bdcc057d41"
Jul 16 22:38:34.047538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:34.047422    2555 scope.go:117] "RemoveContainer" containerID="081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7"
Jul 16 22:38:34.047717 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:34.047694    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs8ht_odh-ai-gateway-infra(43ff5046-263e-401d-8d6d-03993600d26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" podUID="43ff5046-263e-401d-8d6d-03993600d26a"
Jul 16 22:38:35.055017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:35.054988    2555 scope.go:117] "RemoveContainer" containerID="081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7"
Jul 16 22:38:35.055417 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:35.055160    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs8ht_odh-ai-gateway-infra(43ff5046-263e-401d-8d6d-03993600d26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" podUID="43ff5046-263e-401d-8d6d-03993600d26a"
Jul 16 22:38:35.191409 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:35.191370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:38:35.673047 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:35.673011    2555 scope.go:117] "RemoveContainer" containerID="25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3"
Jul 16 22:38:36.062537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:36.062502    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerStarted","Data":"8e0389ac8c16483b1a6b5cb42823164682118e6e6fac7d0ad75ceabefd5d04a6"}
Jul 16 22:38:36.106697 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:36.106644    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" podStartSLOduration=16.106626458 podStartE2EDuration="16.106626458s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:36.106405563 +0000 UTC m=+2970.943459363" watchObservedRunningTime="2026-07-16 22:38:36.106626458 +0000 UTC m=+2970.943680258"
Jul 16 22:38:37.069771 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:37.069737    2555 generic.go:356] "Generic (PLEG): container finished" podID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerID="8e0389ac8c16483b1a6b5cb42823164682118e6e6fac7d0ad75ceabefd5d04a6" exitCode=6
Jul 16 22:38:37.070182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:37.069808    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerDied","Data":"8e0389ac8c16483b1a6b5cb42823164682118e6e6fac7d0ad75ceabefd5d04a6"}
Jul 16 22:38:37.070182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:37.069868    2555 scope.go:117] "RemoveContainer" containerID="25dac5b98c01e90fd958676ca452f75281f75ae9408f31c2b4a28622c534eba3"
Jul 16 22:38:37.070182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:37.070151    2555 scope.go:117] "RemoveContainer" containerID="8e0389ac8c16483b1a6b5cb42823164682118e6e6fac7d0ad75ceabefd5d04a6"
Jul 16 22:38:37.070378 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:37.070356    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj_odh-ai-gateway-infra(42e0a2d9-28c0-428c-ad48-e411bc630bb8)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8"
Jul 16 22:38:37.109036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:37.109006    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:38.212605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:38.212571    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:38.279437 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:38.279393    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca\") pod \"42e0a2d9-28c0-428c-ad48-e411bc630bb8\" (UID: \"42e0a2d9-28c0-428c-ad48-e411bc630bb8\") "
Jul 16 22:38:38.279889 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:38.279868    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "42e0a2d9-28c0-428c-ad48-e411bc630bb8" (UID: "42e0a2d9-28c0-428c-ad48-e411bc630bb8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:38.380057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:38.379973    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/42e0a2d9-28c0-428c-ad48-e411bc630bb8-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:38:39.084513 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.084483    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"
Jul 16 22:38:39.084672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.084450    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj" event={"ID":"42e0a2d9-28c0-428c-ad48-e411bc630bb8","Type":"ContainerDied","Data":"0bdcfdbaecb52dec4d61f5c00217805264991677bf2192325e60e5ec38b7b549"}
Jul 16 22:38:39.084672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.084621    2555 scope.go:117] "RemoveContainer" containerID="8e0389ac8c16483b1a6b5cb42823164682118e6e6fac7d0ad75ceabefd5d04a6"
Jul 16 22:38:39.117099 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.117068    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:39.121345 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.121323    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-m2nhj"]
Jul 16 22:38:39.581159 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581119    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581537    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581549    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581558    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581565    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581630    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.581648 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.581640    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:39.585888 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.585870    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:39.596916 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.596777    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:38:39.676260 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.676220    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" path="/var/lib/kubelet/pods/42e0a2d9-28c0-428c-ad48-e411bc630bb8/volumes"
Jul 16 22:38:39.689744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.689716    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-tm482\" (UID: \"a8dcce96-8c5c-4042-989d-4d18aee7ddbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:39.790760 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.790722    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-tm482\" (UID: \"a8dcce96-8c5c-4042-989d-4d18aee7ddbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:39.791309 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.791291    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-tm482\" (UID: \"a8dcce96-8c5c-4042-989d-4d18aee7ddbb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:39.898292 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:39.898206    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:40.042920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:40.042895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:38:40.044841 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:38:40.044810    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8dcce96_8c5c_4042_989d_4d18aee7ddbb.slice/crio-883da8b9f050b4fdd1d2189d0ad8f5644ad703898bcc0f8ad2ce7339659f45c6 WatchSource:0}: Error finding container 883da8b9f050b4fdd1d2189d0ad8f5644ad703898bcc0f8ad2ce7339659f45c6: Status 404 returned error can't find the container with id 883da8b9f050b4fdd1d2189d0ad8f5644ad703898bcc0f8ad2ce7339659f45c6
Jul 16 22:38:40.091414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:40.091380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerStarted","Data":"883da8b9f050b4fdd1d2189d0ad8f5644ad703898bcc0f8ad2ce7339659f45c6"}
Jul 16 22:38:41.099750 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:41.099716    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerID="54bcdca756f02dfe59116220cf7a1a0b1ed5f2aeda346d651a9c1d8d9ae8b720" exitCode=6
Jul 16 22:38:41.100200 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:41.099769    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerDied","Data":"54bcdca756f02dfe59116220cf7a1a0b1ed5f2aeda346d651a9c1d8d9ae8b720"}
Jul 16 22:38:41.100200 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:41.099972    2555 scope.go:117] "RemoveContainer" containerID="54bcdca756f02dfe59116220cf7a1a0b1ed5f2aeda346d651a9c1d8d9ae8b720"
Jul 16 22:38:42.107331 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:42.107294    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerID="cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470" exitCode=6
Jul 16 22:38:42.107767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:42.107385    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerDied","Data":"cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470"}
Jul 16 22:38:42.107767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:42.107451    2555 scope.go:117] "RemoveContainer" containerID="54bcdca756f02dfe59116220cf7a1a0b1ed5f2aeda346d651a9c1d8d9ae8b720"
Jul 16 22:38:42.107767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:42.107615    2555 scope.go:117] "RemoveContainer" containerID="cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470"
Jul 16 22:38:42.107894 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:42.107838    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-tm482_odh-ai-gateway-infra(a8dcce96-8c5c-4042-989d-4d18aee7ddbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb"
Jul 16 22:38:43.120335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:43.120307    2555 scope.go:117] "RemoveContainer" containerID="cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470"
Jul 16 22:38:43.120763 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:43.120524    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-tm482_odh-ai-gateway-infra(a8dcce96-8c5c-4042-989d-4d18aee7ddbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb"
Jul 16 22:38:44.484625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:44.484585    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:38:48.670901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:48.670871    2555 scope.go:117] "RemoveContainer" containerID="081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7"
Jul 16 22:38:49.157779 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.157740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerStarted","Data":"c4a8d454e1b62c225d62fc77f3ec5c5bd174fd59bbb36f4ba884b1948a17ebdd"}
Jul 16 22:38:49.543634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.543599    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:38:49.543961 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.543950    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:49.544009 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.543963    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:49.544046 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.544031    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="42e0a2d9-28c0-428c-ad48-e411bc630bb8" containerName="revoke-keys"
Jul 16 22:38:49.547098 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.547075    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:38:49.560934 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.560867    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:38:49.680618 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.680588    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85\" (UID: \"12a1d992-0a8d-4f75-8808-e31ce13c5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:38:49.781975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.781936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85\" (UID: \"12a1d992-0a8d-4f75-8808-e31ce13c5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:38:49.782598 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.782581    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85\" (UID: \"12a1d992-0a8d-4f75-8808-e31ce13c5d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:38:49.857441 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.857365    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:38:49.997311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:49.997283    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:38:49.998830 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:38:49.998805    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12a1d992_0a8d_4f75_8808_e31ce13c5d9a.slice/crio-f490230e3ed819667e870a8b06fd8468c72d6a98edde2cf568aee457cb950a41 WatchSource:0}: Error finding container f490230e3ed819667e870a8b06fd8468c72d6a98edde2cf568aee457cb950a41: Status 404 returned error can't find the container with id f490230e3ed819667e870a8b06fd8468c72d6a98edde2cf568aee457cb950a41
Jul 16 22:38:50.165150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.165118    2555 generic.go:356] "Generic (PLEG): container finished" podID="43ff5046-263e-401d-8d6d-03993600d26a" containerID="c4a8d454e1b62c225d62fc77f3ec5c5bd174fd59bbb36f4ba884b1948a17ebdd" exitCode=6
Jul 16 22:38:50.165314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.165190    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerDied","Data":"c4a8d454e1b62c225d62fc77f3ec5c5bd174fd59bbb36f4ba884b1948a17ebdd"}
Jul 16 22:38:50.165314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.165249    2555 scope.go:117] "RemoveContainer" containerID="081ba2183ffd15f42e39ef404fc823c05e8af7c72e75e5bf6abd6daf4d8bacd7"
Jul 16 22:38:50.165581 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.165561    2555 scope.go:117] "RemoveContainer" containerID="c4a8d454e1b62c225d62fc77f3ec5c5bd174fd59bbb36f4ba884b1948a17ebdd"
Jul 16 22:38:50.165843 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:50.165823    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-zs8ht_odh-ai-gateway-infra(43ff5046-263e-401d-8d6d-03993600d26a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" podUID="43ff5046-263e-401d-8d6d-03993600d26a"
Jul 16 22:38:50.166517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.166494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerStarted","Data":"f490230e3ed819667e870a8b06fd8468c72d6a98edde2cf568aee457cb950a41"}
Jul 16 22:38:50.191644 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:50.191614    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:51.175339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.175304    2555 generic.go:356] "Generic (PLEG): container finished" podID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerID="8ce606ad3e6ad361a062383ad864dbe86a49bb4cffe13a9ab0f1f5cb24d61e2a" exitCode=6
Jul 16 22:38:51.175775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.175358    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerDied","Data":"8ce606ad3e6ad361a062383ad864dbe86a49bb4cffe13a9ab0f1f5cb24d61e2a"}
Jul 16 22:38:51.175775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.175591    2555 scope.go:117] "RemoveContainer" containerID="8ce606ad3e6ad361a062383ad864dbe86a49bb4cffe13a9ab0f1f5cb24d61e2a"
Jul 16 22:38:51.306856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.306833    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:51.397425 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.397393    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca\") pod \"43ff5046-263e-401d-8d6d-03993600d26a\" (UID: \"43ff5046-263e-401d-8d6d-03993600d26a\") "
Jul 16 22:38:51.397790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.397766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "43ff5046-263e-401d-8d6d-03993600d26a" (UID: "43ff5046-263e-401d-8d6d-03993600d26a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:51.499034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:51.498946    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/43ff5046-263e-401d-8d6d-03993600d26a-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:38:52.182385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.182290    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"
Jul 16 22:38:52.182385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.182328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht" event={"ID":"43ff5046-263e-401d-8d6d-03993600d26a","Type":"ContainerDied","Data":"008d21b79e37355e27246fc9e70734019a44ccd4d15f28b83c1e68d524f5618c"}
Jul 16 22:38:52.182954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.182405    2555 scope.go:117] "RemoveContainer" containerID="c4a8d454e1b62c225d62fc77f3ec5c5bd174fd59bbb36f4ba884b1948a17ebdd"
Jul 16 22:38:52.184532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.184484    2555 generic.go:356] "Generic (PLEG): container finished" podID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerID="6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739" exitCode=6
Jul 16 22:38:52.184656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.184562    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerDied","Data":"6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739"}
Jul 16 22:38:52.184924 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.184902    2555 scope.go:117] "RemoveContainer" containerID="6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739"
Jul 16 22:38:52.185198 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:52.185174    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85_odh-ai-gateway-infra(12a1d992-0a8d-4f75-8808-e31ce13c5d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a"
Jul 16 22:38:52.191945 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.191924    2555 scope.go:117] "RemoveContainer" containerID="8ce606ad3e6ad361a062383ad864dbe86a49bb4cffe13a9ab0f1f5cb24d61e2a"
Jul 16 22:38:52.215792 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.215760    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:52.221701 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:52.221674    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-zs8ht"]
Jul 16 22:38:53.193286 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:53.193206    2555 scope.go:117] "RemoveContainer" containerID="6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739"
Jul 16 22:38:53.193681 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:53.193449    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85_odh-ai-gateway-infra(12a1d992-0a8d-4f75-8808-e31ce13c5d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a"
Jul 16 22:38:53.676780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:53.676746    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43ff5046-263e-401d-8d6d-03993600d26a" path="/var/lib/kubelet/pods/43ff5046-263e-401d-8d6d-03993600d26a/volumes"
Jul 16 22:38:56.670385 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:56.670358    2555 scope.go:117] "RemoveContainer" containerID="cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470"
Jul 16 22:38:57.217426 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:57.217346    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerStarted","Data":"94de41fe2b254457dc356112843d42e976a3d32de9a4af57e53fb79d9aa8f24a"}
Jul 16 22:38:57.242299 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:57.242247    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" podStartSLOduration=18.242232937 podStartE2EDuration="18.242232937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:38:57.241126577 +0000 UTC m=+2992.078180377" watchObservedRunningTime="2026-07-16 22:38:57.242232937 +0000 UTC m=+2992.079286744"
Jul 16 22:38:58.224335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:58.224297    2555 generic.go:356] "Generic (PLEG): container finished" podID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerID="94de41fe2b254457dc356112843d42e976a3d32de9a4af57e53fb79d9aa8f24a" exitCode=6
Jul 16 22:38:58.224793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:58.224368    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerDied","Data":"94de41fe2b254457dc356112843d42e976a3d32de9a4af57e53fb79d9aa8f24a"}
Jul 16 22:38:58.224793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:58.224426    2555 scope.go:117] "RemoveContainer" containerID="cb73e9211e2acb13ce279157e4e4891b0f7f9e73df0eb8d4b20628c1a973b470"
Jul 16 22:38:58.224793 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:58.224687    2555 scope.go:117] "RemoveContainer" containerID="94de41fe2b254457dc356112843d42e976a3d32de9a4af57e53fb79d9aa8f24a"
Jul 16 22:38:58.224955 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:38:58.224901    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-tm482_odh-ai-gateway-infra(a8dcce96-8c5c-4042-989d-4d18aee7ddbb)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb"
Jul 16 22:38:58.257731 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:58.257702    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:38:59.371148 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:59.371122    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:38:59.472153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:59.472126    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca\") pod \"a8dcce96-8c5c-4042-989d-4d18aee7ddbb\" (UID: \"a8dcce96-8c5c-4042-989d-4d18aee7ddbb\") "
Jul 16 22:38:59.472548 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:59.472519    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a8dcce96-8c5c-4042-989d-4d18aee7ddbb" (UID: "a8dcce96-8c5c-4042-989d-4d18aee7ddbb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:38:59.573165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:38:59.573136    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8dcce96-8c5c-4042-989d-4d18aee7ddbb-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:39:00.241000 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:00.240968    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"
Jul 16 22:39:00.241195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:00.241004    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482" event={"ID":"a8dcce96-8c5c-4042-989d-4d18aee7ddbb","Type":"ContainerDied","Data":"883da8b9f050b4fdd1d2189d0ad8f5644ad703898bcc0f8ad2ce7339659f45c6"}
Jul 16 22:39:00.241195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:00.241075    2555 scope.go:117] "RemoveContainer" containerID="94de41fe2b254457dc356112843d42e976a3d32de9a4af57e53fb79d9aa8f24a"
Jul 16 22:39:00.276009 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:00.275981    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:39:00.282973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:00.282949    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-tm482"]
Jul 16 22:39:01.676934 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:01.676902    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" path="/var/lib/kubelet/pods/a8dcce96-8c5c-4042-989d-4d18aee7ddbb/volumes"
Jul 16 22:39:03.713107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:03.713070    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:39:06.671109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:06.671076    2555 scope.go:117] "RemoveContainer" containerID="6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739"
Jul 16 22:39:07.284030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:07.283996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerStarted","Data":"6bf34ce5df37f49c89a71d0679f07b067794c50db35b40aff01922915a1ba3be"}
Jul 16 22:39:07.332688 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:07.332646    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" podStartSLOduration=18.33262329 podStartE2EDuration="18.33262329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:38:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:07.331652221 +0000 UTC m=+3002.168706022" watchObservedRunningTime="2026-07-16 22:39:07.33262329 +0000 UTC m=+3002.169677148"
Jul 16 22:39:08.291713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:08.291671    2555 generic.go:356] "Generic (PLEG): container finished" podID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerID="6bf34ce5df37f49c89a71d0679f07b067794c50db35b40aff01922915a1ba3be" exitCode=6
Jul 16 22:39:08.292202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:08.291747    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerDied","Data":"6bf34ce5df37f49c89a71d0679f07b067794c50db35b40aff01922915a1ba3be"}
Jul 16 22:39:08.292202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:08.291833    2555 scope.go:117] "RemoveContainer" containerID="6832bad7f5c89815ea2b25a490bd141bf94875184c557b97643c4723cefbf739"
Jul 16 22:39:08.292202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:08.292105    2555 scope.go:117] "RemoveContainer" containerID="6bf34ce5df37f49c89a71d0679f07b067794c50db35b40aff01922915a1ba3be"
Jul 16 22:39:08.292400 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:08.292376    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85_odh-ai-gateway-infra(12a1d992-0a8d-4f75-8808-e31ce13c5d9a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a"
Jul 16 22:39:08.339811 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:08.339786    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:39:09.435752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:09.435730    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:39:09.561499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:09.561364    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca\") pod \"12a1d992-0a8d-4f75-8808-e31ce13c5d9a\" (UID: \"12a1d992-0a8d-4f75-8808-e31ce13c5d9a\") "
Jul 16 22:39:09.561716 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:09.561695    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12a1d992-0a8d-4f75-8808-e31ce13c5d9a" (UID: "12a1d992-0a8d-4f75-8808-e31ce13c5d9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:09.662817 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:09.662781    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12a1d992-0a8d-4f75-8808-e31ce13c5d9a-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:39:10.305513 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.305457    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"
Jul 16 22:39:10.305513 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.305492    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85" event={"ID":"12a1d992-0a8d-4f75-8808-e31ce13c5d9a","Type":"ContainerDied","Data":"f490230e3ed819667e870a8b06fd8468c72d6a98edde2cf568aee457cb950a41"}
Jul 16 22:39:10.305787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.305565    2555 scope.go:117] "RemoveContainer" containerID="6bf34ce5df37f49c89a71d0679f07b067794c50db35b40aff01922915a1ba3be"
Jul 16 22:39:10.324875 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.324845    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:39:10.333112 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.333082    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-bjh85"]
Jul 16 22:39:10.763298 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763212    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763596    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763611    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763623    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763629    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763638    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763647    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763659    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763665    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763679    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763684    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763689    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763694    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763699    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.763699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763704    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763767    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763776    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763781    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763786    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763794    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:10.764152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.763802    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:10.768153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.768133    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:10.771237 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.771214    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:39:10.771380 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.771359    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:39:10.781140 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.781118    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:10.873519 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.873454    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p\" (UID: \"f08ac26d-1e72-436f-ab0e-bdeaea9d012d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:10.974373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.974338    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p\" (UID: \"f08ac26d-1e72-436f-ab0e-bdeaea9d012d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:10.974920 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:10.974901    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p\" (UID: \"f08ac26d-1e72-436f-ab0e-bdeaea9d012d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:11.079177 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:11.079142    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:11.239673 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:11.239643    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:11.240525 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:39:11.240499    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf08ac26d_1e72_436f_ab0e_bdeaea9d012d.slice/crio-71615ab3fa8067f983206924d1ad49570f7cfa9455dc27b6b65a8815da9f319c WatchSource:0}: Error finding container 71615ab3fa8067f983206924d1ad49570f7cfa9455dc27b6b65a8815da9f319c: Status 404 returned error can't find the container with id 71615ab3fa8067f983206924d1ad49570f7cfa9455dc27b6b65a8815da9f319c
Jul 16 22:39:11.312328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:11.312297    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" event={"ID":"f08ac26d-1e72-436f-ab0e-bdeaea9d012d","Type":"ContainerStarted","Data":"71615ab3fa8067f983206924d1ad49570f7cfa9455dc27b6b65a8815da9f319c"}
Jul 16 22:39:11.676975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:11.676891    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" path="/var/lib/kubelet/pods/12a1d992-0a8d-4f75-8808-e31ce13c5d9a/volumes"
Jul 16 22:39:12.319507 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:12.319448    2555 generic.go:356] "Generic (PLEG): container finished" podID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerID="6d04adb48b3a4422c6ae71291d915a3b01723f42f4aa036a6060cdc5746ce279" exitCode=6
Jul 16 22:39:12.319901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:12.319511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" event={"ID":"f08ac26d-1e72-436f-ab0e-bdeaea9d012d","Type":"ContainerDied","Data":"6d04adb48b3a4422c6ae71291d915a3b01723f42f4aa036a6060cdc5746ce279"}
Jul 16 22:39:12.319901 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:12.319747    2555 scope.go:117] "RemoveContainer" containerID="6d04adb48b3a4422c6ae71291d915a3b01723f42f4aa036a6060cdc5746ce279"
Jul 16 22:39:13.326602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:13.326569    2555 generic.go:356] "Generic (PLEG): container finished" podID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerID="b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee" exitCode=6
Jul 16 22:39:13.327023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:13.326659    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" event={"ID":"f08ac26d-1e72-436f-ab0e-bdeaea9d012d","Type":"ContainerDied","Data":"b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee"}
Jul 16 22:39:13.327023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:13.326743    2555 scope.go:117] "RemoveContainer" containerID="6d04adb48b3a4422c6ae71291d915a3b01723f42f4aa036a6060cdc5746ce279"
Jul 16 22:39:13.327023 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:13.326938    2555 scope.go:117] "RemoveContainer" containerID="b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee"
Jul 16 22:39:13.327194 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:13.327175    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p_odh-ai-gateway-infra(f08ac26d-1e72-436f-ab0e-bdeaea9d012d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d"
Jul 16 22:39:14.333904 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:14.333875    2555 scope.go:117] "RemoveContainer" containerID="b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee"
Jul 16 22:39:14.334264 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:14.334039    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p_odh-ai-gateway-infra(f08ac26d-1e72-436f-ab0e-bdeaea9d012d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d"
Jul 16 22:39:25.674093 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:25.674055    2555 scope.go:117] "RemoveContainer" containerID="b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee"
Jul 16 22:39:26.404176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:26.404135    2555 generic.go:356] "Generic (PLEG): container finished" podID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerID="789f109540cac85375446a9f528fed9440b1769c91268743b6b62278fda121d8" exitCode=6
Jul 16 22:39:26.404339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:26.404211    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" event={"ID":"f08ac26d-1e72-436f-ab0e-bdeaea9d012d","Type":"ContainerDied","Data":"789f109540cac85375446a9f528fed9440b1769c91268743b6b62278fda121d8"}
Jul 16 22:39:26.404339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:26.404300    2555 scope.go:117] "RemoveContainer" containerID="b316da6821c72dab9ff6da65ef3c7f23ee7b873230b57a606ce035f9857c2eee"
Jul 16 22:39:26.404663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:26.404642    2555 scope.go:117] "RemoveContainer" containerID="789f109540cac85375446a9f528fed9440b1769c91268743b6b62278fda121d8"
Jul 16 22:39:26.404846 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:26.404827    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p_odh-ai-gateway-infra(f08ac26d-1e72-436f-ab0e-bdeaea9d012d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d"
Jul 16 22:39:27.434147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:27.434112    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:27.567120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:27.567093    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:27.609314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:27.609282    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca\") pod \"f08ac26d-1e72-436f-ab0e-bdeaea9d012d\" (UID: \"f08ac26d-1e72-436f-ab0e-bdeaea9d012d\") "
Jul 16 22:39:27.609661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:27.609637    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f08ac26d-1e72-436f-ab0e-bdeaea9d012d" (UID: "f08ac26d-1e72-436f-ab0e-bdeaea9d012d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:27.710678 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:27.710605    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f08ac26d-1e72-436f-ab0e-bdeaea9d012d-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:39:28.418224 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:28.418197    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"
Jul 16 22:39:28.418408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:28.418238    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p" event={"ID":"f08ac26d-1e72-436f-ab0e-bdeaea9d012d","Type":"ContainerDied","Data":"71615ab3fa8067f983206924d1ad49570f7cfa9455dc27b6b65a8815da9f319c"}
Jul 16 22:39:28.418408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:28.418303    2555 scope.go:117] "RemoveContainer" containerID="789f109540cac85375446a9f528fed9440b1769c91268743b6b62278fda121d8"
Jul 16 22:39:28.443211 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:28.443188    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:28.453860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:28.453838    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4cr7p"]
Jul 16 22:39:29.676073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:29.676029    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" path="/var/lib/kubelet/pods/f08ac26d-1e72-436f-ab0e-bdeaea9d012d/volumes"
Jul 16 22:39:31.335861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.335825    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:31.336358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336340    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336362    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336377    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336386    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336402    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336411    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.336430 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336426    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336434    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336539    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336552    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="12a1d992-0a8d-4f75-8808-e31ce13c5d9a" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336563    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="43ff5046-263e-401d-8d6d-03993600d26a" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336573    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8dcce96-8c5c-4042-989d-4d18aee7ddbb" containerName="revoke-keys"
Jul 16 22:39:31.336773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.336583    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:31.343546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.343516    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:31.346807 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.346782    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:39:31.347032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.347005    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:39:31.349417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.349394    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:31.444776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.444743    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ds575\" (UID: \"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:31.545253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.545218    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ds575\" (UID: \"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:31.545843 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.545823    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-ds575\" (UID: \"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:31.656564 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.656489    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:31.795301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:31.795267    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:31.797161 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:39:31.797134    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2c8d469_0e03_4fb1_8a53_5fc0ec94c461.slice/crio-50d0344643f22b0c747f7aa3365fc72e7eec0d369880166acca397850353e1cf WatchSource:0}: Error finding container 50d0344643f22b0c747f7aa3365fc72e7eec0d369880166acca397850353e1cf: Status 404 returned error can't find the container with id 50d0344643f22b0c747f7aa3365fc72e7eec0d369880166acca397850353e1cf
Jul 16 22:39:32.442503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:32.442454    2555 generic.go:356] "Generic (PLEG): container finished" podID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerID="1fd3755438535d4da59302cf5bf4d8e74e23b9f911d98dd9e83bf85ae2bad5d7" exitCode=6
Jul 16 22:39:32.442862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:32.442493    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" event={"ID":"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461","Type":"ContainerDied","Data":"1fd3755438535d4da59302cf5bf4d8e74e23b9f911d98dd9e83bf85ae2bad5d7"}
Jul 16 22:39:32.442862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:32.442553    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" event={"ID":"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461","Type":"ContainerStarted","Data":"50d0344643f22b0c747f7aa3365fc72e7eec0d369880166acca397850353e1cf"}
Jul 16 22:39:32.442862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:32.442757    2555 scope.go:117] "RemoveContainer" containerID="1fd3755438535d4da59302cf5bf4d8e74e23b9f911d98dd9e83bf85ae2bad5d7"
Jul 16 22:39:33.449605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:33.449569    2555 generic.go:356] "Generic (PLEG): container finished" podID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerID="6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7" exitCode=6
Jul 16 22:39:33.450063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:33.449655    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" event={"ID":"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461","Type":"ContainerDied","Data":"6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7"}
Jul 16 22:39:33.450063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:33.449728    2555 scope.go:117] "RemoveContainer" containerID="1fd3755438535d4da59302cf5bf4d8e74e23b9f911d98dd9e83bf85ae2bad5d7"
Jul 16 22:39:33.450063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:33.449915    2555 scope.go:117] "RemoveContainer" containerID="6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7"
Jul 16 22:39:33.450191 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:33.450108    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ds575_odh-ai-gateway-infra(b2c8d469-0e03-4fb1-8a53-5fc0ec94c461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461"
Jul 16 22:39:34.457348 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:34.457319    2555 scope.go:117] "RemoveContainer" containerID="6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7"
Jul 16 22:39:34.457859 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:34.457584    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ds575_odh-ai-gateway-infra(b2c8d469-0e03-4fb1-8a53-5fc0ec94c461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461"
Jul 16 22:39:39.631618 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.631576    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:39:39.632113 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.632095    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:39.632157 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.632117    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:39.632256 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.632243    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f08ac26d-1e72-436f-ab0e-bdeaea9d012d" containerName="revoke-keys"
Jul 16 22:39:39.636802 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.636781    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:39.646492 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.646433    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:39:39.715282 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.715246    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd\" (UID: \"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:39.816097 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.816061    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd\" (UID: \"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:39.816778 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.816758    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd\" (UID: \"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:39.948923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:39.948849    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:40.089511 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:40.089484    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:39:40.091711 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:39:40.091682    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff7e51e_e28f_4d34_8c2f_7f1e4726c88c.slice/crio-1af902023d4dbc5f21760a3889af25e2797d241938fa9c98f9ede9f02e8444d0 WatchSource:0}: Error finding container 1af902023d4dbc5f21760a3889af25e2797d241938fa9c98f9ede9f02e8444d0: Status 404 returned error can't find the container with id 1af902023d4dbc5f21760a3889af25e2797d241938fa9c98f9ede9f02e8444d0
Jul 16 22:39:40.490276 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:40.490110    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerStarted","Data":"dc54308e86dd368cbf68ea77ebcd4dd9baecf83bfbc48d4a1d5dbd36667ec0d4"}
Jul 16 22:39:40.490276 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:40.490170    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerStarted","Data":"1af902023d4dbc5f21760a3889af25e2797d241938fa9c98f9ede9f02e8444d0"}
Jul 16 22:39:41.497455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:41.497424    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerID="dc54308e86dd368cbf68ea77ebcd4dd9baecf83bfbc48d4a1d5dbd36667ec0d4" exitCode=6
Jul 16 22:39:41.497879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:41.497498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerDied","Data":"dc54308e86dd368cbf68ea77ebcd4dd9baecf83bfbc48d4a1d5dbd36667ec0d4"}
Jul 16 22:39:41.497879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:41.497705    2555 scope.go:117] "RemoveContainer" containerID="dc54308e86dd368cbf68ea77ebcd4dd9baecf83bfbc48d4a1d5dbd36667ec0d4"
Jul 16 22:39:42.505276 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:42.505176    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerID="ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e" exitCode=6
Jul 16 22:39:42.505276 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:42.505253    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerDied","Data":"ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e"}
Jul 16 22:39:42.505821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:42.505314    2555 scope.go:117] "RemoveContainer" containerID="dc54308e86dd368cbf68ea77ebcd4dd9baecf83bfbc48d4a1d5dbd36667ec0d4"
Jul 16 22:39:42.505821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:42.505530    2555 scope.go:117] "RemoveContainer" containerID="ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e"
Jul 16 22:39:42.505821 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:42.505737    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd_odh-ai-gateway-infra(8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c"
Jul 16 22:39:43.513301 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:43.513275    2555 scope.go:117] "RemoveContainer" containerID="ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e"
Jul 16 22:39:43.513743 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:43.513445    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd_odh-ai-gateway-infra(8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c"
Jul 16 22:39:49.630010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.629980    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:39:49.633365 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.633350    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:39:49.647390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.647359    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:39:49.671277 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.671253    2555 scope.go:117] "RemoveContainer" containerID="6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7"
Jul 16 22:39:49.707528 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.707502    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd\" (UID: \"539fd1c0-19cf-4706-9831-077840997821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:39:49.809544 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.808936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd\" (UID: \"539fd1c0-19cf-4706-9831-077840997821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:39:49.809728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.809699    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd\" (UID: \"539fd1c0-19cf-4706-9831-077840997821\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:39:49.943338 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:49.943309    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:39:50.093859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.093836    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:39:50.095685 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:39:50.095658    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod539fd1c0_19cf_4706_9831_077840997821.slice/crio-386623890f08f71ed5258f2541b713ba6417a80c69049f35ae34f05755021613 WatchSource:0}: Error finding container 386623890f08f71ed5258f2541b713ba6417a80c69049f35ae34f05755021613: Status 404 returned error can't find the container with id 386623890f08f71ed5258f2541b713ba6417a80c69049f35ae34f05755021613
Jul 16 22:39:50.555044 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.555011    2555 generic.go:356] "Generic (PLEG): container finished" podID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerID="9b9df963030ff84b4357cdd3df25f07d59fef6631c6b0f04eafb8ab7421c709b" exitCode=6
Jul 16 22:39:50.555223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.555087    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" event={"ID":"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461","Type":"ContainerDied","Data":"9b9df963030ff84b4357cdd3df25f07d59fef6631c6b0f04eafb8ab7421c709b"}
Jul 16 22:39:50.555223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.555147    2555 scope.go:117] "RemoveContainer" containerID="6ee8c2a659f167c59cbfeb601ca615ef01617a49d18716950cfccbe29e0efbd7"
Jul 16 22:39:50.555554 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.555531    2555 scope.go:117] "RemoveContainer" containerID="9b9df963030ff84b4357cdd3df25f07d59fef6631c6b0f04eafb8ab7421c709b"
Jul 16 22:39:50.555792 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:50.555770    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-ds575_odh-ai-gateway-infra(b2c8d469-0e03-4fb1-8a53-5fc0ec94c461)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461"
Jul 16 22:39:50.556756 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.556737    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerStarted","Data":"cb54a3b002fe52b1c1f6c03c84e15684fcdbd8b9f464cc8e54a407a2c63cd5f6"}
Jul 16 22:39:50.556863 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.556781    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerStarted","Data":"386623890f08f71ed5258f2541b713ba6417a80c69049f35ae34f05755021613"}
Jul 16 22:39:50.624858 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:50.624750    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" podStartSLOduration=1.624733633 podStartE2EDuration="1.624733633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:50.623070831 +0000 UTC m=+3045.460124631" watchObservedRunningTime="2026-07-16 22:39:50.624733633 +0000 UTC m=+3045.461787433"
Jul 16 22:39:51.565443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.565407    2555 generic.go:356] "Generic (PLEG): container finished" podID="539fd1c0-19cf-4706-9831-077840997821" containerID="cb54a3b002fe52b1c1f6c03c84e15684fcdbd8b9f464cc8e54a407a2c63cd5f6" exitCode=6
Jul 16 22:39:51.565859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.565494    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerDied","Data":"cb54a3b002fe52b1c1f6c03c84e15684fcdbd8b9f464cc8e54a407a2c63cd5f6"}
Jul 16 22:39:51.565859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.565669    2555 scope.go:117] "RemoveContainer" containerID="cb54a3b002fe52b1c1f6c03c84e15684fcdbd8b9f464cc8e54a407a2c63cd5f6"
Jul 16 22:39:51.603198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.603169    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:51.743684 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.743662    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:51.827410 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.827340    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca\") pod \"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461\" (UID: \"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461\") "
Jul 16 22:39:51.827728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.827707    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" (UID: "b2c8d469-0e03-4fb1-8a53-5fc0ec94c461"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:39:51.928129 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:51.928103    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:39:52.573044 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.573011    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"
Jul 16 22:39:52.573505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.573011    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575" event={"ID":"b2c8d469-0e03-4fb1-8a53-5fc0ec94c461","Type":"ContainerDied","Data":"50d0344643f22b0c747f7aa3365fc72e7eec0d369880166acca397850353e1cf"}
Jul 16 22:39:52.573505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.573162    2555 scope.go:117] "RemoveContainer" containerID="9b9df963030ff84b4357cdd3df25f07d59fef6631c6b0f04eafb8ab7421c709b"
Jul 16 22:39:52.575014 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.574994    2555 generic.go:356] "Generic (PLEG): container finished" podID="539fd1c0-19cf-4706-9831-077840997821" containerID="05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff" exitCode=6
Jul 16 22:39:52.575107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.575049    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerDied","Data":"05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff"}
Jul 16 22:39:52.575261 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.575247    2555 scope.go:117] "RemoveContainer" containerID="05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff"
Jul 16 22:39:52.575445 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:52.575425    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd_odh-ai-gateway-infra(539fd1c0-19cf-4706-9831-077840997821)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" podUID="539fd1c0-19cf-4706-9831-077840997821"
Jul 16 22:39:52.583360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.583344    2555 scope.go:117] "RemoveContainer" containerID="cb54a3b002fe52b1c1f6c03c84e15684fcdbd8b9f464cc8e54a407a2c63cd5f6"
Jul 16 22:39:52.607003 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.606978    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:52.612844 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:52.612825    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-ds575"]
Jul 16 22:39:53.583789 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.583764    2555 scope.go:117] "RemoveContainer" containerID="05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff"
Jul 16 22:39:53.584154 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:53.583941    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd_odh-ai-gateway-infra(539fd1c0-19cf-4706-9831-077840997821)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" podUID="539fd1c0-19cf-4706-9831-077840997821"
Jul 16 22:39:53.676703 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.676668    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" path="/var/lib/kubelet/pods/b2c8d469-0e03-4fb1-8a53-5fc0ec94c461/volumes"
Jul 16 22:39:53.712476 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712438    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:39:53.712866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712848    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.712954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712867    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.712954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712898    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.712954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712904    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.713057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712982    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.713057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.712993    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:39:53.717376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.717353    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.721428 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.721409    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-dbc614\"/\"default-dockercfg-z677v\""
Jul 16 22:39:53.721832 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.721813    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-dbc614\"/\"auth-test-model-dbc614-kserve-self-signed-certs\""
Jul 16 22:39:53.721832 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.721827    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-dbc614\"/\"kube-root-ca.crt\""
Jul 16 22:39:53.721966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.721872    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-dbc614\"/\"openshift-service-ca.crt\""
Jul 16 22:39:53.731088 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.731063    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:39:53.845744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845660    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcmjt\" (UniqueName: \"kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.845744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845708    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.845744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845732    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.845958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845826    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.845958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845888    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.846030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.845961    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.946967    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947013    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcmjt\" (UniqueName: \"kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947040    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947068    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947108    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947140    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947540 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947517    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947574    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.947602 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.947568    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.949342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.949318    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.949746 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.949727    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:53.955941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:53.955923    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcmjt\" (UniqueName: \"kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt\") pod \"auth-test-model-dbc614-kserve-565b7bcfff-w46ws\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") " pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:54.026025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:54.025997    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:39:54.165958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:54.165925    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:39:54.166572 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:39:54.166531    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29f292c_3571_45a1_a900_131051d02496.slice/crio-20d7b8912a47fafdcdff0e0e1c122636d5e45b2226fcf333423bdd0fea26e5b5 WatchSource:0}: Error finding container 20d7b8912a47fafdcdff0e0e1c122636d5e45b2226fcf333423bdd0fea26e5b5: Status 404 returned error can't find the container with id 20d7b8912a47fafdcdff0e0e1c122636d5e45b2226fcf333423bdd0fea26e5b5
Jul 16 22:39:54.590935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:54.590899    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" event={"ID":"c29f292c-3571-45a1-a900-131051d02496","Type":"ContainerStarted","Data":"1289c4e02d50d2043b24452558e621f0317552efb3447af2eb7c3a071bf42b63"}
Jul 16 22:39:54.590935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:54.590941    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" event={"ID":"c29f292c-3571-45a1-a900-131051d02496","Type":"ContainerStarted","Data":"20d7b8912a47fafdcdff0e0e1c122636d5e45b2226fcf333423bdd0fea26e5b5"}
Jul 16 22:39:54.609653 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:54.609607    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" podStartSLOduration=1.609593364 podStartE2EDuration="1.609593364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:39:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:39:54.608306847 +0000 UTC m=+3049.445360648" watchObservedRunningTime="2026-07-16 22:39:54.609593364 +0000 UTC m=+3049.446647164"
Jul 16 22:39:57.483496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:57.483394    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:39:57.670279 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:57.670252    2555 scope.go:117] "RemoveContainer" containerID="ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e"
Jul 16 22:39:58.617505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:58.617403    2555 generic.go:356] "Generic (PLEG): container finished" podID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerID="4ca826ea7a343f8d1dad7420b23f4a4962d1a85b6abc0f1433f788376134d1c9" exitCode=6
Jul 16 22:39:58.617505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:58.617458    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerDied","Data":"4ca826ea7a343f8d1dad7420b23f4a4962d1a85b6abc0f1433f788376134d1c9"}
Jul 16 22:39:58.618026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:58.617531    2555 scope.go:117] "RemoveContainer" containerID="ae10fffab1334c0091df83f4d4e29f35efeecff418782b90555001f1fa3f3e0e"
Jul 16 22:39:58.618026 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:58.617839    2555 scope.go:117] "RemoveContainer" containerID="4ca826ea7a343f8d1dad7420b23f4a4962d1a85b6abc0f1433f788376134d1c9"
Jul 16 22:39:58.618133 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:39:58.618105    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd_odh-ai-gateway-infra(8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c"
Jul 16 22:39:59.651167 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:59.651137    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:39:59.787649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:59.787621    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:39:59.899011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:59.898979    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca\") pod \"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c\" (UID: \"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c\") "
Jul 16 22:39:59.899335 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:39:59.899312    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" (UID: "8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:00.000386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.000300    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:40:00.631579 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.631549    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"
Jul 16 22:40:00.631781 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.631597    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd" event={"ID":"8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c","Type":"ContainerDied","Data":"1af902023d4dbc5f21760a3889af25e2797d241938fa9c98f9ede9f02e8444d0"}
Jul 16 22:40:00.631781 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.631657    2555 scope.go:117] "RemoveContainer" containerID="4ca826ea7a343f8d1dad7420b23f4a4962d1a85b6abc0f1433f788376134d1c9"
Jul 16 22:40:00.661622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.661584    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:40:00.667580 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.667553    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7g9rd"]
Jul 16 22:40:00.812958 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.812925    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:00.813293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813281    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813295    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813303    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813308    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813324    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:40:00.813343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813330    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:40:00.813525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813383    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813391    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2c8d469-0e03-4fb1-8a53-5fc0ec94c461" containerName="revoke-keys"
Jul 16 22:40:00.813525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813397    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.813525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.813403    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:00.817946 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.817931    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:00.827394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.827370    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:00.906912 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:00.906830    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br\" (UID: \"130465e8-1571-4e11-a53f-1433db4025cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:01.007741 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.007695    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br\" (UID: \"130465e8-1571-4e11-a53f-1433db4025cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:01.008340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.008321    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br\" (UID: \"130465e8-1571-4e11-a53f-1433db4025cd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:01.129810 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.129774    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:01.281699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.279219    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:01.638453 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.638410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerStarted","Data":"b06cb680faf21001df8cb94fb03f0b00e898139ddc21e269940d2cfaef3c1f90"}
Jul 16 22:40:01.638678 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.638507    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerStarted","Data":"518b6caa264c15239969790fea593ca85bee2c42278d7fd637f1ab3fd9613d43"}
Jul 16 22:40:01.663999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.663950    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" podStartSLOduration=1.663931973 podStartE2EDuration="1.663931973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:01.661522786 +0000 UTC m=+3056.498576586" watchObservedRunningTime="2026-07-16 22:40:01.663931973 +0000 UTC m=+3056.500985775"
Jul 16 22:40:01.677736 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:01.677703    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" path="/var/lib/kubelet/pods/8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c/volumes"
Jul 16 22:40:02.647735 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:02.647695    2555 generic.go:356] "Generic (PLEG): container finished" podID="130465e8-1571-4e11-a53f-1433db4025cd" containerID="b06cb680faf21001df8cb94fb03f0b00e898139ddc21e269940d2cfaef3c1f90" exitCode=6
Jul 16 22:40:02.647937 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:02.647778    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerDied","Data":"b06cb680faf21001df8cb94fb03f0b00e898139ddc21e269940d2cfaef3c1f90"}
Jul 16 22:40:02.648008 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:02.647934    2555 scope.go:117] "RemoveContainer" containerID="b06cb680faf21001df8cb94fb03f0b00e898139ddc21e269940d2cfaef3c1f90"
Jul 16 22:40:03.655850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:03.655762    2555 generic.go:356] "Generic (PLEG): container finished" podID="130465e8-1571-4e11-a53f-1433db4025cd" containerID="e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7" exitCode=6
Jul 16 22:40:03.655850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:03.655806    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerDied","Data":"e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7"}
Jul 16 22:40:03.656359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:03.655866    2555 scope.go:117] "RemoveContainer" containerID="b06cb680faf21001df8cb94fb03f0b00e898139ddc21e269940d2cfaef3c1f90"
Jul 16 22:40:03.656359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:03.656053    2555 scope.go:117] "RemoveContainer" containerID="e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7"
Jul 16 22:40:03.656359 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:03.656319    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br_odh-ai-gateway-infra(130465e8-1571-4e11-a53f-1433db4025cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" podUID="130465e8-1571-4e11-a53f-1433db4025cd"
Jul 16 22:40:04.026879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:04.026853    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:40:04.027048 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:04.026898    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:40:04.039320 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:04.039286    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:40:04.663172 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:04.663144    2555 scope.go:117] "RemoveContainer" containerID="e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7"
Jul 16 22:40:04.663682 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:04.663353    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br_odh-ai-gateway-infra(130465e8-1571-4e11-a53f-1433db4025cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" podUID="130465e8-1571-4e11-a53f-1433db4025cd"
Jul 16 22:40:04.674605 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:04.674575    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:40:06.671119 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.671084    2555 scope.go:117] "RemoveContainer" containerID="05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff"
Jul 16 22:40:06.687349 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.687314    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:06.687741 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.687727    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:06.687788 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.687745    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ff7e51e-e28f-4d34-8c2f-7f1e4726c88c" containerName="revoke-keys"
Jul 16 22:40:06.692601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.692580    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:06.694914 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.694898    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-xfrn6\""
Jul 16 22:40:06.699128 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.699108    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:06.863424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.863392    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rt6f\" (UniqueName: \"kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f\") pod \"kuadrant-operator-catalog-qrf9c\" (UID: \"614f0055-3748-49c8-9a74-a80bed53bf09\") " pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:06.965016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.964928    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rt6f\" (UniqueName: \"kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f\") pod \"kuadrant-operator-catalog-qrf9c\" (UID: \"614f0055-3748-49c8-9a74-a80bed53bf09\") " pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:06.999263 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:06.999244    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rt6f\" (UniqueName: \"kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f\") pod \"kuadrant-operator-catalog-qrf9c\" (UID: \"614f0055-3748-49c8-9a74-a80bed53bf09\") " pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:07.019179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.019155    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:07.156688 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.156660    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:07.158131 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:40:07.158103    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod614f0055_3748_49c8_9a74_a80bed53bf09.slice/crio-f49fcdf98dc69e40ff2be487f4bb8c0d0e7590074be30d1981dd5ebf6b56349a WatchSource:0}: Error finding container f49fcdf98dc69e40ff2be487f4bb8c0d0e7590074be30d1981dd5ebf6b56349a: Status 404 returned error can't find the container with id f49fcdf98dc69e40ff2be487f4bb8c0d0e7590074be30d1981dd5ebf6b56349a
Jul 16 22:40:07.159667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.159650    2555 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:40:07.684436 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.684356    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" event={"ID":"614f0055-3748-49c8-9a74-a80bed53bf09","Type":"ContainerStarted","Data":"f49fcdf98dc69e40ff2be487f4bb8c0d0e7590074be30d1981dd5ebf6b56349a"}
Jul 16 22:40:07.687183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.687154    2555 generic.go:356] "Generic (PLEG): container finished" podID="539fd1c0-19cf-4706-9831-077840997821" containerID="d8edec17938e0479b4c7040bf8d4b8747c4979731bbc08594e5476ea7c16eb03" exitCode=6
Jul 16 22:40:07.687303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.687177    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerDied","Data":"d8edec17938e0479b4c7040bf8d4b8747c4979731bbc08594e5476ea7c16eb03"}
Jul 16 22:40:07.687303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.687248    2555 scope.go:117] "RemoveContainer" containerID="05fe0dd8adabc43970acfdb1dfe7c6b0d0c3ad3469c77a4cf063f735b39f47ff"
Jul 16 22:40:07.687678 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:07.687658    2555 scope.go:117] "RemoveContainer" containerID="d8edec17938e0479b4c7040bf8d4b8747c4979731bbc08594e5476ea7c16eb03"
Jul 16 22:40:07.687937 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:07.687886    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd_odh-ai-gateway-infra(539fd1c0-19cf-4706-9831-077840997821)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" podUID="539fd1c0-19cf-4706-9831-077840997821"
Jul 16 22:40:08.715499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:08.715447    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:40:09.040229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.040194    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:40:09.186924 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.186894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca\") pod \"539fd1c0-19cf-4706-9831-077840997821\" (UID: \"539fd1c0-19cf-4706-9831-077840997821\") "
Jul 16 22:40:09.187243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.187215    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "539fd1c0-19cf-4706-9831-077840997821" (UID: "539fd1c0-19cf-4706-9831-077840997821"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:09.288424 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.288396    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/539fd1c0-19cf-4706-9831-077840997821-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:40:09.701747 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.701717    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" event={"ID":"614f0055-3748-49c8-9a74-a80bed53bf09","Type":"ContainerStarted","Data":"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"}
Jul 16 22:40:09.702995 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.702978    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"
Jul 16 22:40:09.703104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.703042    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd" event={"ID":"539fd1c0-19cf-4706-9831-077840997821","Type":"ContainerDied","Data":"386623890f08f71ed5258f2541b713ba6417a80c69049f35ae34f05755021613"}
Jul 16 22:40:09.703104 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.703093    2555 scope.go:117] "RemoveContainer" containerID="d8edec17938e0479b4c7040bf8d4b8747c4979731bbc08594e5476ea7c16eb03"
Jul 16 22:40:09.726176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.726129    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" podStartSLOduration=1.768803323 podStartE2EDuration="3.726114323s" totalImagesPullingTime="1.957311s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:09.724141689 +0000 UTC m=+3064.561195499" watchObservedRunningTime="2026-07-16 22:40:09.726114323 +0000 UTC m=+3064.563168123"
Jul 16 22:40:09.746122 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.746087    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:40:09.751826 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:09.751800    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-swlwd"]
Jul 16 22:40:11.676689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:11.676653    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="539fd1c0-19cf-4706-9831-077840997821" path="/var/lib/kubelet/pods/539fd1c0-19cf-4706-9831-077840997821/volumes"
Jul 16 22:40:11.902248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:11.902209    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:40:15.673822 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:15.673791    2555 scope.go:117] "RemoveContainer" containerID="e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7"
Jul 16 22:40:16.746336 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:16.746247    2555 generic.go:356] "Generic (PLEG): container finished" podID="130465e8-1571-4e11-a53f-1433db4025cd" containerID="d138075261c25018a924b40a489576d291e5b0a8174e78b15790d6b229271737" exitCode=6
Jul 16 22:40:16.746336 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:16.746287    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerDied","Data":"d138075261c25018a924b40a489576d291e5b0a8174e78b15790d6b229271737"}
Jul 16 22:40:16.746833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:16.746348    2555 scope.go:117] "RemoveContainer" containerID="e223cfb8d1c2e6c1077a2443c19ee02057bdd255e6f09f75fa01a0e1c19fd8a7"
Jul 16 22:40:16.746833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:16.746671    2555 scope.go:117] "RemoveContainer" containerID="d138075261c25018a924b40a489576d291e5b0a8174e78b15790d6b229271737"
Jul 16 22:40:16.746940 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:16.746918    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br_odh-ai-gateway-infra(130465e8-1571-4e11-a53f-1433db4025cd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" podUID="130465e8-1571-4e11-a53f-1433db4025cd"
Jul 16 22:40:17.019745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.019641    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:17.019745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.019713    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:17.042566 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.042541    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:17.775842 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.775813    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:17.799093 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.799067    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:17.824903 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.824873    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:17.932531 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.932509    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:17.965422 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.965399    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca\") pod \"130465e8-1571-4e11-a53f-1433db4025cd\" (UID: \"130465e8-1571-4e11-a53f-1433db4025cd\") "
Jul 16 22:40:17.965753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:17.965732    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "130465e8-1571-4e11-a53f-1433db4025cd" (UID: "130465e8-1571-4e11-a53f-1433db4025cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:18.066199 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.066161    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/130465e8-1571-4e11-a53f-1433db4025cd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:40:18.759909 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.759879    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"
Jul 16 22:40:18.760079 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.759912    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br" event={"ID":"130465e8-1571-4e11-a53f-1433db4025cd","Type":"ContainerDied","Data":"518b6caa264c15239969790fea593ca85bee2c42278d7fd637f1ab3fd9613d43"}
Jul 16 22:40:18.760079 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.759968    2555 scope.go:117] "RemoveContainer" containerID="d138075261c25018a924b40a489576d291e5b0a8174e78b15790d6b229271737"
Jul 16 22:40:18.783175 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.783147    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:18.789664 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:18.789644    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-hm6br"]
Jul 16 22:40:19.676539 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:19.676508    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="130465e8-1571-4e11-a53f-1433db4025cd" path="/var/lib/kubelet/pods/130465e8-1571-4e11-a53f-1433db4025cd/volumes"
Jul 16 22:40:19.766578 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:19.766542    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" podUID="614f0055-3748-49c8-9a74-a80bed53bf09" containerName="registry-server" containerID="cri-o://ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648" gracePeriod=2
Jul 16 22:40:20.020827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.020805    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:20.081668 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.081638    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9rt6f\" (UniqueName: \"kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f\") pod \"614f0055-3748-49c8-9a74-a80bed53bf09\" (UID: \"614f0055-3748-49c8-9a74-a80bed53bf09\") "
Jul 16 22:40:20.083823 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.083795    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f" (OuterVolumeSpecName: "kube-api-access-9rt6f") pod "614f0055-3748-49c8-9a74-a80bed53bf09" (UID: "614f0055-3748-49c8-9a74-a80bed53bf09"). InnerVolumeSpecName "kube-api-access-9rt6f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:20.183229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.183151    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rt6f\" (UniqueName: \"kubernetes.io/projected/614f0055-3748-49c8-9a74-a80bed53bf09-kube-api-access-9rt6f\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:40:20.772952 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.772915    2555 generic.go:356] "Generic (PLEG): container finished" podID="614f0055-3748-49c8-9a74-a80bed53bf09" containerID="ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648" exitCode=0
Jul 16 22:40:20.773109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.772971    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c"
Jul 16 22:40:20.773109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.772970    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" event={"ID":"614f0055-3748-49c8-9a74-a80bed53bf09","Type":"ContainerDied","Data":"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"}
Jul 16 22:40:20.773109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.773046    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-qrf9c" event={"ID":"614f0055-3748-49c8-9a74-a80bed53bf09","Type":"ContainerDied","Data":"f49fcdf98dc69e40ff2be487f4bb8c0d0e7590074be30d1981dd5ebf6b56349a"}
Jul 16 22:40:20.773109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.773073    2555 scope.go:117] "RemoveContainer" containerID="ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"
Jul 16 22:40:20.782783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.782768    2555 scope.go:117] "RemoveContainer" containerID="ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"
Jul 16 22:40:20.783029 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:20.783010    2555 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648\": container with ID starting with ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648 not found: ID does not exist" containerID="ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"
Jul 16 22:40:20.783073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.783039    2555 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648"} err="failed to get container status \"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648\": rpc error: code = NotFound desc = could not find container \"ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648\": container with ID starting with ed9175e67b1b21757a5fbf88623e78267377eb925eda37f271be415373864648 not found: ID does not exist"
Jul 16 22:40:20.797582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.797552    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:20.807951 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:20.807918    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-qrf9c"]
Jul 16 22:40:21.675860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:21.675827    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="614f0055-3748-49c8-9a74-a80bed53bf09" path="/var/lib/kubelet/pods/614f0055-3748-49c8-9a74-a80bed53bf09/volumes"
Jul 16 22:40:25.797651 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:25.797617    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:40:31.413018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.412980    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413327    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="614f0055-3748-49c8-9a74-a80bed53bf09" containerName="registry-server"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413338    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="614f0055-3748-49c8-9a74-a80bed53bf09" containerName="registry-server"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413346    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413351    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413361    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413366    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413373    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413378    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413384    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413389    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413396    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413401    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413409    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413414    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413477    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413486    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413495    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413502    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:31.413525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.413511    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="614f0055-3748-49c8-9a74-a80bed53bf09" containerName="registry-server"
Jul 16 22:40:31.418053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.418036    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:31.420526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.420491    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:40:31.420633 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.420524    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:40:31.425070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.424908    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:31.477942 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.477902    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-8jfkn\" (UID: \"a920a60a-48bd-4910-b8a9-b155cec6ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:31.578601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.578571    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-8jfkn\" (UID: \"a920a60a-48bd-4910-b8a9-b155cec6ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:31.579205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.579186    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-8jfkn\" (UID: \"a920a60a-48bd-4910-b8a9-b155cec6ab07\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:31.730928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.730841    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:31.870214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:31.870189    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:31.871995 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:40:31.871968    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda920a60a_48bd_4910_b8a9_b155cec6ab07.slice/crio-1a05568187af3cbfc036831458d4df43f19b346984fa8dc92aae3539e6430126 WatchSource:0}: Error finding container 1a05568187af3cbfc036831458d4df43f19b346984fa8dc92aae3539e6430126: Status 404 returned error can't find the container with id 1a05568187af3cbfc036831458d4df43f19b346984fa8dc92aae3539e6430126
Jul 16 22:40:32.845746 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:32.845707    2555 generic.go:356] "Generic (PLEG): container finished" podID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerID="45dff696e52e7c96165877dc1809bdccc4403533c521c9a8c5bdab7ff2b1c019" exitCode=6
Jul 16 22:40:32.846137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:32.845773    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" event={"ID":"a920a60a-48bd-4910-b8a9-b155cec6ab07","Type":"ContainerDied","Data":"45dff696e52e7c96165877dc1809bdccc4403533c521c9a8c5bdab7ff2b1c019"}
Jul 16 22:40:32.846137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:32.845827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" event={"ID":"a920a60a-48bd-4910-b8a9-b155cec6ab07","Type":"ContainerStarted","Data":"1a05568187af3cbfc036831458d4df43f19b346984fa8dc92aae3539e6430126"}
Jul 16 22:40:32.846137 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:32.846030    2555 scope.go:117] "RemoveContainer" containerID="45dff696e52e7c96165877dc1809bdccc4403533c521c9a8c5bdab7ff2b1c019"
Jul 16 22:40:33.853311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:33.853273    2555 generic.go:356] "Generic (PLEG): container finished" podID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerID="b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b" exitCode=6
Jul 16 22:40:33.853758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:33.853357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" event={"ID":"a920a60a-48bd-4910-b8a9-b155cec6ab07","Type":"ContainerDied","Data":"b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b"}
Jul 16 22:40:33.853758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:33.853423    2555 scope.go:117] "RemoveContainer" containerID="45dff696e52e7c96165877dc1809bdccc4403533c521c9a8c5bdab7ff2b1c019"
Jul 16 22:40:33.853758 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:33.853548    2555 scope.go:117] "RemoveContainer" containerID="b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b"
Jul 16 22:40:33.853918 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:33.853770    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-8jfkn_odh-ai-gateway-infra(a920a60a-48bd-4910-b8a9-b155cec6ab07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07"
Jul 16 22:40:34.860845 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:34.860819    2555 scope.go:117] "RemoveContainer" containerID="b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b"
Jul 16 22:40:34.861212 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:34.861036    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-8jfkn_odh-ai-gateway-infra(a920a60a-48bd-4910-b8a9-b155cec6ab07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07"
Jul 16 22:40:38.885223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:38.885192    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:40:39.687541 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.687508    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:40:39.687932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.687918    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="539fd1c0-19cf-4706-9831-077840997821" containerName="revoke-keys"
Jul 16 22:40:39.688012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.687936    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="130465e8-1571-4e11-a53f-1433db4025cd" containerName="revoke-keys"
Jul 16 22:40:39.692489 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.692435    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:40:39.699300 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.699256    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:40:39.752246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.752206    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj\" (UID: \"6d79797e-2c05-4f1b-9411-0fe5626357f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:40:39.853123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.853090    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj\" (UID: \"6d79797e-2c05-4f1b-9411-0fe5626357f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:40:39.853686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:39.853668    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj\" (UID: \"6d79797e-2c05-4f1b-9411-0fe5626357f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:40:40.004458 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.004359    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:40:40.142622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.142601    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:40:40.144498 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:40:40.144454    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d79797e_2c05_4f1b_9411_0fe5626357f2.slice/crio-bf007ae76a9cbdf30a0cca6363f6d974c0f1b89199f40226f7d53cc68b0b81fe WatchSource:0}: Error finding container bf007ae76a9cbdf30a0cca6363f6d974c0f1b89199f40226f7d53cc68b0b81fe: Status 404 returned error can't find the container with id bf007ae76a9cbdf30a0cca6363f6d974c0f1b89199f40226f7d53cc68b0b81fe
Jul 16 22:40:40.900528 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.900493    2555 generic.go:356] "Generic (PLEG): container finished" podID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerID="e45566d72e4ec77a13c0868ade2b0063d577a8faa12d6cf615144d69a6809ad0" exitCode=6
Jul 16 22:40:40.900720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.900574    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerDied","Data":"e45566d72e4ec77a13c0868ade2b0063d577a8faa12d6cf615144d69a6809ad0"}
Jul 16 22:40:40.900720 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.900664    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerStarted","Data":"bf007ae76a9cbdf30a0cca6363f6d974c0f1b89199f40226f7d53cc68b0b81fe"}
Jul 16 22:40:40.900826 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:40.900776    2555 scope.go:117] "RemoveContainer" containerID="e45566d72e4ec77a13c0868ade2b0063d577a8faa12d6cf615144d69a6809ad0"
Jul 16 22:40:41.908741 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:41.908704    2555 generic.go:356] "Generic (PLEG): container finished" podID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerID="08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82" exitCode=6
Jul 16 22:40:41.909176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:41.908788    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerDied","Data":"08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82"}
Jul 16 22:40:41.909176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:41.908840    2555 scope.go:117] "RemoveContainer" containerID="e45566d72e4ec77a13c0868ade2b0063d577a8faa12d6cf615144d69a6809ad0"
Jul 16 22:40:41.909176 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:41.909016    2555 scope.go:117] "RemoveContainer" containerID="08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82"
Jul 16 22:40:41.909354 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:41.909270    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj_odh-ai-gateway-infra(6d79797e-2c05-4f1b-9411-0fe5626357f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2"
Jul 16 22:40:42.917067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:42.917036    2555 scope.go:117] "RemoveContainer" containerID="08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82"
Jul 16 22:40:42.917560 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:42.917282    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj_odh-ai-gateway-infra(6d79797e-2c05-4f1b-9411-0fe5626357f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2"
Jul 16 22:40:46.671161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:46.671122    2555 scope.go:117] "RemoveContainer" containerID="b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b"
Jul 16 22:40:47.948849 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:47.948813    2555 generic.go:356] "Generic (PLEG): container finished" podID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerID="02964d42d07066adfc05e14f06e83a02bd5bce7588db103a6df1514fb962cfba" exitCode=6
Jul 16 22:40:47.949334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:47.948888    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" event={"ID":"a920a60a-48bd-4910-b8a9-b155cec6ab07","Type":"ContainerDied","Data":"02964d42d07066adfc05e14f06e83a02bd5bce7588db103a6df1514fb962cfba"}
Jul 16 22:40:47.949334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:47.948950    2555 scope.go:117] "RemoveContainer" containerID="b64e6af1f76807725ab7d06c9434bc88932e3a206f8e3dab54bc0baf191b7a0b"
Jul 16 22:40:47.949334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:47.949239    2555 scope.go:117] "RemoveContainer" containerID="02964d42d07066adfc05e14f06e83a02bd5bce7588db103a6df1514fb962cfba"
Jul 16 22:40:47.949491 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:47.949437    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-8jfkn_odh-ai-gateway-infra(a920a60a-48bd-4910-b8a9-b155cec6ab07)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07"
Jul 16 22:40:48.977284 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:48.977250    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:49.116280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.116251    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:49.239947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.239849    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca\") pod \"a920a60a-48bd-4910-b8a9-b155cec6ab07\" (UID: \"a920a60a-48bd-4910-b8a9-b155cec6ab07\") "
Jul 16 22:40:49.240205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.240182    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a920a60a-48bd-4910-b8a9-b155cec6ab07" (UID: "a920a60a-48bd-4910-b8a9-b155cec6ab07"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:49.341475 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.341436    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a920a60a-48bd-4910-b8a9-b155cec6ab07-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:40:49.693764 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.693732    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:40:49.694076 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694065    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.694123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694078    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.694123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694088    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.694123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694094    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.694236 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694152    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.694236 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.694162    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:49.697566 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.697548    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:40:49.705317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.705287    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:40:49.846398 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.846356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-prf24\" (UID: \"ef7c715b-9159-42a4-b1a1-8456ce2cf52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:40:49.947202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.947124    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-prf24\" (UID: \"ef7c715b-9159-42a4-b1a1-8456ce2cf52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:40:49.947786 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.947769    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-prf24\" (UID: \"ef7c715b-9159-42a4-b1a1-8456ce2cf52e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:40:49.963021 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.962994    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"
Jul 16 22:40:49.963165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.963034    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn" event={"ID":"a920a60a-48bd-4910-b8a9-b155cec6ab07","Type":"ContainerDied","Data":"1a05568187af3cbfc036831458d4df43f19b346984fa8dc92aae3539e6430126"}
Jul 16 22:40:49.963165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.963097    2555 scope.go:117] "RemoveContainer" containerID="02964d42d07066adfc05e14f06e83a02bd5bce7588db103a6df1514fb962cfba"
Jul 16 22:40:49.980774 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.980749    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:49.987342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:49.987320    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-8jfkn"]
Jul 16 22:40:50.008936 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.008912    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:40:50.352962 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.352938    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:40:50.857420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.857383    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:40:50.857809 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.857795    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:50.857860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.857811    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:50.857898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.857877    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" containerName="revoke-keys"
Jul 16 22:40:50.860948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.860933    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:40:50.868351 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.868316    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:40:50.956193 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.956167    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85\" (UID: \"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:40:50.971059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.971034    2555 generic.go:356] "Generic (PLEG): container finished" podID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerID="1e417ef6808782341c553a177eef4862eecef6d97f2384c2ef00070eb1d1e58b" exitCode=6
Jul 16 22:40:50.971190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.971113    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerDied","Data":"1e417ef6808782341c553a177eef4862eecef6d97f2384c2ef00070eb1d1e58b"}
Jul 16 22:40:50.971190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.971149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerStarted","Data":"b1062adad5f0ee9159a85107ccbc0f3ca1ce2769d8f5bdf8e704892be7ae7e1f"}
Jul 16 22:40:50.971287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:50.971272    2555 scope.go:117] "RemoveContainer" containerID="1e417ef6808782341c553a177eef4862eecef6d97f2384c2ef00070eb1d1e58b"
Jul 16 22:40:51.057530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.057494    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85\" (UID: \"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:40:51.058079 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.058058    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85\" (UID: \"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:40:51.173248 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.173159    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:40:51.309893 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.309868    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:40:51.310838 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:40:51.310806    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddcd8035_e4b9_43e2_bdbb_3f2c61959f7c.slice/crio-100a18e4cd2064ec22a454659ccf00dfb860bd6b8e7bf68526ae0cc3f26c09ac WatchSource:0}: Error finding container 100a18e4cd2064ec22a454659ccf00dfb860bd6b8e7bf68526ae0cc3f26c09ac: Status 404 returned error can't find the container with id 100a18e4cd2064ec22a454659ccf00dfb860bd6b8e7bf68526ae0cc3f26c09ac
Jul 16 22:40:51.677133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.677060    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a920a60a-48bd-4910-b8a9-b155cec6ab07" path="/var/lib/kubelet/pods/a920a60a-48bd-4910-b8a9-b155cec6ab07/volumes"
Jul 16 22:40:51.979411 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.979330    2555 generic.go:356] "Generic (PLEG): container finished" podID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerID="06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc" exitCode=6
Jul 16 22:40:51.979590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.979411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerDied","Data":"06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc"}
Jul 16 22:40:51.979590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.979490    2555 scope.go:117] "RemoveContainer" containerID="1e417ef6808782341c553a177eef4862eecef6d97f2384c2ef00070eb1d1e58b"
Jul 16 22:40:51.979716 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.979701    2555 scope.go:117] "RemoveContainer" containerID="06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc"
Jul 16 22:40:51.979946 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:51.979925    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-prf24_odh-ai-gateway-infra(ef7c715b-9159-42a4-b1a1-8456ce2cf52e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e"
Jul 16 22:40:51.981100 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.981081    2555 generic.go:356] "Generic (PLEG): container finished" podID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerID="2ff8b36c022a4a134097fd28d892771ab6c7458edf77361f415d75bc054388bc" exitCode=6
Jul 16 22:40:51.981208 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.981167    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerDied","Data":"2ff8b36c022a4a134097fd28d892771ab6c7458edf77361f415d75bc054388bc"}
Jul 16 22:40:51.981273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.981216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerStarted","Data":"100a18e4cd2064ec22a454659ccf00dfb860bd6b8e7bf68526ae0cc3f26c09ac"}
Jul 16 22:40:51.981273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:51.981269    2555 scope.go:117] "RemoveContainer" containerID="2ff8b36c022a4a134097fd28d892771ab6c7458edf77361f415d75bc054388bc"
Jul 16 22:40:52.989591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:52.989489    2555 scope.go:117] "RemoveContainer" containerID="06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc"
Jul 16 22:40:52.989945 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:52.989726    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-prf24_odh-ai-gateway-infra(ef7c715b-9159-42a4-b1a1-8456ce2cf52e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e"
Jul 16 22:40:52.991131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:52.991107    2555 generic.go:356] "Generic (PLEG): container finished" podID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerID="f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701" exitCode=6
Jul 16 22:40:52.991265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:52.991189    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerDied","Data":"f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701"}
Jul 16 22:40:52.991265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:52.991244    2555 scope.go:117] "RemoveContainer" containerID="2ff8b36c022a4a134097fd28d892771ab6c7458edf77361f415d75bc054388bc"
Jul 16 22:40:52.991387 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:52.991376    2555 scope.go:117] "RemoveContainer" containerID="f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701"
Jul 16 22:40:52.991606 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:52.991586    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85_odh-ai-gateway-infra(ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c"
Jul 16 22:40:53.695973 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:53.695946    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:40:53.999017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:53.998936    2555 scope.go:117] "RemoveContainer" containerID="f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701"
Jul 16 22:40:53.999374 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:40:53.999153    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85_odh-ai-gateway-infra(ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c"
Jul 16 22:40:58.670553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:58.670518    2555 scope.go:117] "RemoveContainer" containerID="08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82"
Jul 16 22:40:59.027624 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:59.027599    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerStarted","Data":"bea84294bc46d500798b3dbe99d8fa1e867ba240098b2d29ebe4ce2394aa59fa"}
Jul 16 22:40:59.050904 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:40:59.050851    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" podStartSLOduration=20.05083223 podStartE2EDuration="20.05083223s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:59.049922551 +0000 UTC m=+3113.886976351" watchObservedRunningTime="2026-07-16 22:40:59.05083223 +0000 UTC m=+3113.887886034"
Jul 16 22:41:00.035559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:00.035520    2555 generic.go:356] "Generic (PLEG): container finished" podID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerID="bea84294bc46d500798b3dbe99d8fa1e867ba240098b2d29ebe4ce2394aa59fa" exitCode=6
Jul 16 22:41:00.035986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:00.035572    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerDied","Data":"bea84294bc46d500798b3dbe99d8fa1e867ba240098b2d29ebe4ce2394aa59fa"}
Jul 16 22:41:00.035986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:00.035646    2555 scope.go:117] "RemoveContainer" containerID="08acbb82984f70ef330f4e44ff796aff968396b2d0fe608234cb05259091bc82"
Jul 16 22:41:00.035986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:00.035906    2555 scope.go:117] "RemoveContainer" containerID="bea84294bc46d500798b3dbe99d8fa1e867ba240098b2d29ebe4ce2394aa59fa"
Jul 16 22:41:00.036150 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:00.036124    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj_odh-ai-gateway-infra(6d79797e-2c05-4f1b-9411-0fe5626357f2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2"
Jul 16 22:41:00.062063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:00.062030    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:41:01.173310 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:01.173288    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:41:01.348404 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:01.348311    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca\") pod \"6d79797e-2c05-4f1b-9411-0fe5626357f2\" (UID: \"6d79797e-2c05-4f1b-9411-0fe5626357f2\") "
Jul 16 22:41:01.348721 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:01.348697    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6d79797e-2c05-4f1b-9411-0fe5626357f2" (UID: "6d79797e-2c05-4f1b-9411-0fe5626357f2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:01.449559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:01.449522    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6d79797e-2c05-4f1b-9411-0fe5626357f2-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:02.050222 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:02.050183    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj" event={"ID":"6d79797e-2c05-4f1b-9411-0fe5626357f2","Type":"ContainerDied","Data":"bf007ae76a9cbdf30a0cca6363f6d974c0f1b89199f40226f7d53cc68b0b81fe"}
Jul 16 22:41:02.050413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:02.050250    2555 scope.go:117] "RemoveContainer" containerID="bea84294bc46d500798b3dbe99d8fa1e867ba240098b2d29ebe4ce2394aa59fa"
Jul 16 22:41:02.050413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:02.050203    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"
Jul 16 22:41:02.069729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:02.069703    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:41:02.073338 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:02.073314    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-zthtj"]
Jul 16 22:41:03.676139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:03.676100    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" path="/var/lib/kubelet/pods/6d79797e-2c05-4f1b-9411-0fe5626357f2/volumes"
Jul 16 22:41:05.673563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:05.673513    2555 scope.go:117] "RemoveContainer" containerID="06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc"
Jul 16 22:41:06.076776 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:06.076744    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerStarted","Data":"52bd32eef6bfb03f0debd8c93fb4d1170cd159fa8d83199ccc1649efe36a26e5"}
Jul 16 22:41:06.093121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:06.093065    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" podStartSLOduration=17.093044581 podStartE2EDuration="17.093044581s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:06.092181898 +0000 UTC m=+3120.929235698" watchObservedRunningTime="2026-07-16 22:41:06.093044581 +0000 UTC m=+3120.930098383"
Jul 16 22:41:07.084772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:07.084740    2555 generic.go:356] "Generic (PLEG): container finished" podID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerID="52bd32eef6bfb03f0debd8c93fb4d1170cd159fa8d83199ccc1649efe36a26e5" exitCode=6
Jul 16 22:41:07.085215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:07.084787    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerDied","Data":"52bd32eef6bfb03f0debd8c93fb4d1170cd159fa8d83199ccc1649efe36a26e5"}
Jul 16 22:41:07.085215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:07.084835    2555 scope.go:117] "RemoveContainer" containerID="06263006e45bad8f7fddf30c662c402bcecb791185701043d3df5101be59bcbc"
Jul 16 22:41:07.085215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:07.085061    2555 scope.go:117] "RemoveContainer" containerID="52bd32eef6bfb03f0debd8c93fb4d1170cd159fa8d83199ccc1649efe36a26e5"
Jul 16 22:41:07.085372 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:07.085292    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-prf24_odh-ai-gateway-infra(ef7c715b-9159-42a4-b1a1-8456ce2cf52e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e"
Jul 16 22:41:07.105814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:07.105780    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:41:08.223775 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.223752    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:41:08.303121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.303087    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca\") pod \"ef7c715b-9159-42a4-b1a1-8456ce2cf52e\" (UID: \"ef7c715b-9159-42a4-b1a1-8456ce2cf52e\") "
Jul 16 22:41:08.303502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.303456    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef7c715b-9159-42a4-b1a1-8456ce2cf52e" (UID: "ef7c715b-9159-42a4-b1a1-8456ce2cf52e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:08.404231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.404147    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef7c715b-9159-42a4-b1a1-8456ce2cf52e-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:08.671383 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.671293    2555 scope.go:117] "RemoveContainer" containerID="f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701"
Jul 16 22:41:08.891273 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:08.891236    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:41:09.102262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.102161    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"
Jul 16 22:41:09.103219 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.102197    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24" event={"ID":"ef7c715b-9159-42a4-b1a1-8456ce2cf52e","Type":"ContainerDied","Data":"b1062adad5f0ee9159a85107ccbc0f3ca1ce2769d8f5bdf8e704892be7ae7e1f"}
Jul 16 22:41:09.103343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.103272    2555 scope.go:117] "RemoveContainer" containerID="52bd32eef6bfb03f0debd8c93fb4d1170cd159fa8d83199ccc1649efe36a26e5"
Jul 16 22:41:09.104951 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.104927    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerStarted","Data":"16fb975097131e50aaeb77af49c38239749cbeed60d49ed28ef4a37e9be95651"}
Jul 16 22:41:09.127371 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.127324    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" podStartSLOduration=19.127310344 podStartE2EDuration="19.127310344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:40:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:09.126499981 +0000 UTC m=+3123.963553777" watchObservedRunningTime="2026-07-16 22:41:09.127310344 +0000 UTC m=+3123.964364143"
Jul 16 22:41:09.143327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.143295    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:41:09.152320 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.152294    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-prf24"]
Jul 16 22:41:09.676535 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:09.676445    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" path="/var/lib/kubelet/pods/ef7c715b-9159-42a4-b1a1-8456ce2cf52e/volumes"
Jul 16 22:41:10.113197 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:10.113156    2555 generic.go:356] "Generic (PLEG): container finished" podID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerID="16fb975097131e50aaeb77af49c38239749cbeed60d49ed28ef4a37e9be95651" exitCode=6
Jul 16 22:41:10.113363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:10.113206    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerDied","Data":"16fb975097131e50aaeb77af49c38239749cbeed60d49ed28ef4a37e9be95651"}
Jul 16 22:41:10.113363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:10.113261    2555 scope.go:117] "RemoveContainer" containerID="f2ff5fa3a1bd28abf665be839fbc98ccbf6e8f1eb92f656925876e144443c701"
Jul 16 22:41:10.113646 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:10.113619    2555 scope.go:117] "RemoveContainer" containerID="16fb975097131e50aaeb77af49c38239749cbeed60d49ed28ef4a37e9be95651"
Jul 16 22:41:10.113903 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:10.113882    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85_odh-ai-gateway-infra(ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c"
Jul 16 22:41:10.142553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:10.142528    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:41:11.256087 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:11.256061    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:41:11.424177 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:11.424084    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca\") pod \"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c\" (UID: \"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c\") "
Jul 16 22:41:11.424503 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:11.424453    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" (UID: "ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:11.524903 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:11.524867    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:12.126412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:12.126381    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"
Jul 16 22:41:12.126412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:12.126392    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85" event={"ID":"ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c","Type":"ContainerDied","Data":"100a18e4cd2064ec22a454659ccf00dfb860bd6b8e7bf68526ae0cc3f26c09ac"}
Jul 16 22:41:12.126670 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:12.126482    2555 scope.go:117] "RemoveContainer" containerID="16fb975097131e50aaeb77af49c38239749cbeed60d49ed28ef4a37e9be95651"
Jul 16 22:41:12.146794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:12.146772    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:41:12.152002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:12.151980    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-j2j85"]
Jul 16 22:41:13.676436 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:13.676403    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" path="/var/lib/kubelet/pods/ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c/volumes"
Jul 16 22:41:21.462146 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462110    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462449    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462476    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462490    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462496    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462503    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462509    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462520    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462526    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462532    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462537    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462544    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462549    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462555    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462560    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462569    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462574    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.462625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462632    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462641    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462646    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462652    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d79797e-2c05-4f1b-9411-0fe5626357f2" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462659    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462665    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462670    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:21.463202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.462676    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:21.465780 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.465764    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:21.468443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.468422    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:41:21.468443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.468420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:41:21.477814 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.477789    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:21.499954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.499916    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-qg7mv\" (UID: \"e575f023-689c-44d4-a7c2-8c884c3a5fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:21.600902 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.600876    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-qg7mv\" (UID: \"e575f023-689c-44d4-a7c2-8c884c3a5fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:21.601499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.601458    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-qg7mv\" (UID: \"e575f023-689c-44d4-a7c2-8c884c3a5fc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:21.777002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.776973    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:21.916163 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:21.916133    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:21.917623 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:41:21.917597    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode575f023_689c_44d4_a7c2_8c884c3a5fc4.slice/crio-b006fca06e95fda5dd804c0edbf2ef7fe7ceb2a2941c839f117012646350a2fb WatchSource:0}: Error finding container b006fca06e95fda5dd804c0edbf2ef7fe7ceb2a2941c839f117012646350a2fb: Status 404 returned error can't find the container with id b006fca06e95fda5dd804c0edbf2ef7fe7ceb2a2941c839f117012646350a2fb
Jul 16 22:41:22.183491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:22.183442    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerStarted","Data":"b006fca06e95fda5dd804c0edbf2ef7fe7ceb2a2941c839f117012646350a2fb"}
Jul 16 22:41:23.190835 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:23.190797    2555 generic.go:356] "Generic (PLEG): container finished" podID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerID="a98e28b06fed46d56670f1db2f66984cc2b491b18e3060db987822f86b9ed704" exitCode=6
Jul 16 22:41:23.191328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:23.190884    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerDied","Data":"a98e28b06fed46d56670f1db2f66984cc2b491b18e3060db987822f86b9ed704"}
Jul 16 22:41:23.191328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:23.191122    2555 scope.go:117] "RemoveContainer" containerID="a98e28b06fed46d56670f1db2f66984cc2b491b18e3060db987822f86b9ed704"
Jul 16 22:41:24.198745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:24.198659    2555 generic.go:356] "Generic (PLEG): container finished" podID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerID="57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64" exitCode=6
Jul 16 22:41:24.198745 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:24.198716    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerDied","Data":"57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64"}
Jul 16 22:41:24.199205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:24.198772    2555 scope.go:117] "RemoveContainer" containerID="a98e28b06fed46d56670f1db2f66984cc2b491b18e3060db987822f86b9ed704"
Jul 16 22:41:24.199205 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:24.198910    2555 scope.go:117] "RemoveContainer" containerID="57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64"
Jul 16 22:41:24.199205 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:24.199178    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-qg7mv_odh-ai-gateway-infra(e575f023-689c-44d4-a7c2-8c884c3a5fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4"
Jul 16 22:41:25.205620 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:25.205588    2555 scope.go:117] "RemoveContainer" containerID="57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64"
Jul 16 22:41:25.206037 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:25.205780    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-qg7mv_odh-ai-gateway-infra(e575f023-689c-44d4-a7c2-8c884c3a5fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4"
Jul 16 22:41:26.793360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:26.793282    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:41:38.671278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:38.671246    2555 scope.go:117] "RemoveContainer" containerID="57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64"
Jul 16 22:41:39.282034 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.282001    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerStarted","Data":"76dc0dd8351f93dfe6d27aadad568fd907b22a37a6c6171dd4508d2a5ef2cef4"}
Jul 16 22:41:39.310537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.310451    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" podStartSLOduration=18.310429361 podStartE2EDuration="18.310429361s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:41:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:41:39.30521069 +0000 UTC m=+3154.142264489" watchObservedRunningTime="2026-07-16 22:41:39.310429361 +0000 UTC m=+3154.147483162"
Jul 16 22:41:39.763923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.763832    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:39.767500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.765015    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:39.767500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.765045    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ddcd8035-e4b9-43e2-bdbb-3f2c61959f7c" containerName="revoke-keys"
Jul 16 22:41:39.767500 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.765258    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef7c715b-9159-42a4-b1a1-8456ce2cf52e" containerName="revoke-keys"
Jul 16 22:41:39.770059 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.770035    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:39.785081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.785047    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:39.802110 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.802079    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:39.805689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.805672    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:39.820413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.820383    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:39.861377 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.861350    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr\" (UID: \"219302cc-e69d-4d11-9747-25e23696e572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:39.861547 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.861389    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7\" (UID: \"7e5c1b8b-9173-4735-930e-d0c338fc3f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:39.962368 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.962337    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr\" (UID: \"219302cc-e69d-4d11-9747-25e23696e572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:39.962565 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.962374    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7\" (UID: \"7e5c1b8b-9173-4735-930e-d0c338fc3f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:39.962946 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.962927    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr\" (UID: \"219302cc-e69d-4d11-9747-25e23696e572\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:39.962985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:39.962958    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7\" (UID: \"7e5c1b8b-9173-4735-930e-d0c338fc3f70\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:40.081290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.081260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:40.116656 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.116623    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:40.237988 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.237930    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:40.239892 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:41:40.239845    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e5c1b8b_9173_4735_930e_d0c338fc3f70.slice/crio-bfa3c4397ac6d1393ef1b9eb072e56312b902d1c9841cdc3d47c3b4251d75d28 WatchSource:0}: Error finding container bfa3c4397ac6d1393ef1b9eb072e56312b902d1c9841cdc3d47c3b4251d75d28: Status 404 returned error can't find the container with id bfa3c4397ac6d1393ef1b9eb072e56312b902d1c9841cdc3d47c3b4251d75d28
Jul 16 22:41:40.275914 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.275747    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:40.276822 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:41:40.276794    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod219302cc_e69d_4d11_9747_25e23696e572.slice/crio-20cdc143d8a41410a51d482c1b2385b808720fb8a0749debea559faaa6360571 WatchSource:0}: Error finding container 20cdc143d8a41410a51d482c1b2385b808720fb8a0749debea559faaa6360571: Status 404 returned error can't find the container with id 20cdc143d8a41410a51d482c1b2385b808720fb8a0749debea559faaa6360571
Jul 16 22:41:40.288506 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.288476    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" event={"ID":"219302cc-e69d-4d11-9747-25e23696e572","Type":"ContainerStarted","Data":"20cdc143d8a41410a51d482c1b2385b808720fb8a0749debea559faaa6360571"}
Jul 16 22:41:40.290307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.290279    2555 generic.go:356] "Generic (PLEG): container finished" podID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerID="76dc0dd8351f93dfe6d27aadad568fd907b22a37a6c6171dd4508d2a5ef2cef4" exitCode=6
Jul 16 22:41:40.290439 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.290362    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerDied","Data":"76dc0dd8351f93dfe6d27aadad568fd907b22a37a6c6171dd4508d2a5ef2cef4"}
Jul 16 22:41:40.290551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.290439    2555 scope.go:117] "RemoveContainer" containerID="57b1df0265a5e184be3475d134abd970765651af82805bd882282c0c3fd7cf64"
Jul 16 22:41:40.290844 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.290822    2555 scope.go:117] "RemoveContainer" containerID="76dc0dd8351f93dfe6d27aadad568fd907b22a37a6c6171dd4508d2a5ef2cef4"
Jul 16 22:41:40.291084 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:40.291057    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-qg7mv_odh-ai-gateway-infra(e575f023-689c-44d4-a7c2-8c884c3a5fc4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4"
Jul 16 22:41:40.291857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.291837    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" event={"ID":"7e5c1b8b-9173-4735-930e-d0c338fc3f70","Type":"ContainerStarted","Data":"bfa3c4397ac6d1393ef1b9eb072e56312b902d1c9841cdc3d47c3b4251d75d28"}
Jul 16 22:41:40.323812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:40.323784    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:41.299884 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.299845    2555 generic.go:356] "Generic (PLEG): container finished" podID="219302cc-e69d-4d11-9747-25e23696e572" containerID="90bbb428e015adcab8d258ca8d69c120282a4f9820eec13e8c13a7681376dee6" exitCode=6
Jul 16 22:41:41.300340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.299932    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" event={"ID":"219302cc-e69d-4d11-9747-25e23696e572","Type":"ContainerDied","Data":"90bbb428e015adcab8d258ca8d69c120282a4f9820eec13e8c13a7681376dee6"}
Jul 16 22:41:41.300340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.300166    2555 scope.go:117] "RemoveContainer" containerID="90bbb428e015adcab8d258ca8d69c120282a4f9820eec13e8c13a7681376dee6"
Jul 16 22:41:41.303088 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.303062    2555 generic.go:356] "Generic (PLEG): container finished" podID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerID="4e77c3fa2bd6d5472c5dfd3ededf5e3147ded028e2fb21aafd80b32d9ff66b85" exitCode=6
Jul 16 22:41:41.303190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.303125    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" event={"ID":"7e5c1b8b-9173-4735-930e-d0c338fc3f70","Type":"ContainerDied","Data":"4e77c3fa2bd6d5472c5dfd3ededf5e3147ded028e2fb21aafd80b32d9ff66b85"}
Jul 16 22:41:41.303337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.303323    2555 scope.go:117] "RemoveContainer" containerID="4e77c3fa2bd6d5472c5dfd3ededf5e3147ded028e2fb21aafd80b32d9ff66b85"
Jul 16 22:41:41.546148 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.546128    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:41.674162 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.674130    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca\") pod \"e575f023-689c-44d4-a7c2-8c884c3a5fc4\" (UID: \"e575f023-689c-44d4-a7c2-8c884c3a5fc4\") "
Jul 16 22:41:41.674584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.674561    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e575f023-689c-44d4-a7c2-8c884c3a5fc4" (UID: "e575f023-689c-44d4-a7c2-8c884c3a5fc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:41.775055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:41.775021    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e575f023-689c-44d4-a7c2-8c884c3a5fc4-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:42.316344 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.316305    2555 generic.go:356] "Generic (PLEG): container finished" podID="219302cc-e69d-4d11-9747-25e23696e572" containerID="48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d" exitCode=6
Jul 16 22:41:42.316857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.316393    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" event={"ID":"219302cc-e69d-4d11-9747-25e23696e572","Type":"ContainerDied","Data":"48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d"}
Jul 16 22:41:42.316857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.316517    2555 scope.go:117] "RemoveContainer" containerID="90bbb428e015adcab8d258ca8d69c120282a4f9820eec13e8c13a7681376dee6"
Jul 16 22:41:42.316857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.316726    2555 scope.go:117] "RemoveContainer" containerID="48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d"
Jul 16 22:41:42.317021 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:42.316941    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr_odh-ai-gateway-infra(219302cc-e69d-4d11-9747-25e23696e572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" podUID="219302cc-e69d-4d11-9747-25e23696e572"
Jul 16 22:41:42.317875 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.317861    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"
Jul 16 22:41:42.317949 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.317870    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv" event={"ID":"e575f023-689c-44d4-a7c2-8c884c3a5fc4","Type":"ContainerDied","Data":"b006fca06e95fda5dd804c0edbf2ef7fe7ceb2a2941c839f117012646350a2fb"}
Jul 16 22:41:42.319827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.319803    2555 generic.go:356] "Generic (PLEG): container finished" podID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerID="34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941" exitCode=6
Jul 16 22:41:42.319970 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.319879    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" event={"ID":"7e5c1b8b-9173-4735-930e-d0c338fc3f70","Type":"ContainerDied","Data":"34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941"}
Jul 16 22:41:42.320155 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.320139    2555 scope.go:117] "RemoveContainer" containerID="34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941"
Jul 16 22:41:42.320396 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:42.320378    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7_odh-ai-gateway-infra(7e5c1b8b-9173-4735-930e-d0c338fc3f70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70"
Jul 16 22:41:42.327947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.327929    2555 scope.go:117] "RemoveContainer" containerID="76dc0dd8351f93dfe6d27aadad568fd907b22a37a6c6171dd4508d2a5ef2cef4"
Jul 16 22:41:42.335558 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.335536    2555 scope.go:117] "RemoveContainer" containerID="4e77c3fa2bd6d5472c5dfd3ededf5e3147ded028e2fb21aafd80b32d9ff66b85"
Jul 16 22:41:42.388249 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.388189    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:42.391999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:42.391975    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-qg7mv"]
Jul 16 22:41:43.327370 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:43.327342    2555 scope.go:117] "RemoveContainer" containerID="48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d"
Jul 16 22:41:43.327857 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:43.327616    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr_odh-ai-gateway-infra(219302cc-e69d-4d11-9747-25e23696e572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" podUID="219302cc-e69d-4d11-9747-25e23696e572"
Jul 16 22:41:43.329727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:43.329709    2555 scope.go:117] "RemoveContainer" containerID="34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941"
Jul 16 22:41:43.329851 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:43.329834    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7_odh-ai-gateway-infra(7e5c1b8b-9173-4735-930e-d0c338fc3f70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70"
Jul 16 22:41:43.678295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:43.678211    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" path="/var/lib/kubelet/pods/e575f023-689c-44d4-a7c2-8c884c3a5fc4/volumes"
Jul 16 22:41:46.093820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:46.093786    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:41:50.903157 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903124    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903527    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903542    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903557    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903562    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903571    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903579    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903651    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903662    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.903828 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.903668    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e575f023-689c-44d4-a7c2-8c884c3a5fc4" containerName="revoke-keys"
Jul 16 22:41:50.906773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.906754    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:41:50.915999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:50.915974    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:41:51.052150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.052117    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6\" (UID: \"aa63dae6-4b4b-4878-abab-d854ecd6035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:41:51.153053 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.153005    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6\" (UID: \"aa63dae6-4b4b-4878-abab-d854ecd6035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:41:51.153709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.153665    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6\" (UID: \"aa63dae6-4b4b-4878-abab-d854ecd6035f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:41:51.218633 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.218607    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:41:51.366429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.366392    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:41:51.366886 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:41:51.366864    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa63dae6_4b4b_4878_abab_d854ecd6035f.slice/crio-4307a30194ea9900b56b240e505b6bc9c8bc99bc6877172098e8b96cbf4805f2 WatchSource:0}: Error finding container 4307a30194ea9900b56b240e505b6bc9c8bc99bc6877172098e8b96cbf4805f2: Status 404 returned error can't find the container with id 4307a30194ea9900b56b240e505b6bc9c8bc99bc6877172098e8b96cbf4805f2
Jul 16 22:41:51.380532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:51.380500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" event={"ID":"aa63dae6-4b4b-4878-abab-d854ecd6035f","Type":"ContainerStarted","Data":"4307a30194ea9900b56b240e505b6bc9c8bc99bc6877172098e8b96cbf4805f2"}
Jul 16 22:41:52.388107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:52.388063    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerID="584079e065c6fb381057d1a451d8a7908820b63fb5fbb0b076725532ba8242ce" exitCode=6
Jul 16 22:41:52.388713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:52.388143    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" event={"ID":"aa63dae6-4b4b-4878-abab-d854ecd6035f","Type":"ContainerDied","Data":"584079e065c6fb381057d1a451d8a7908820b63fb5fbb0b076725532ba8242ce"}
Jul 16 22:41:52.388713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:52.388391    2555 scope.go:117] "RemoveContainer" containerID="584079e065c6fb381057d1a451d8a7908820b63fb5fbb0b076725532ba8242ce"
Jul 16 22:41:53.395810 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:53.395777    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerID="860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b" exitCode=6
Jul 16 22:41:53.396317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:53.395853    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" event={"ID":"aa63dae6-4b4b-4878-abab-d854ecd6035f","Type":"ContainerDied","Data":"860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b"}
Jul 16 22:41:53.396317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:53.395924    2555 scope.go:117] "RemoveContainer" containerID="584079e065c6fb381057d1a451d8a7908820b63fb5fbb0b076725532ba8242ce"
Jul 16 22:41:53.396317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:53.396140    2555 scope.go:117] "RemoveContainer" containerID="860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b"
Jul 16 22:41:53.396532 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:53.396418    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6_odh-ai-gateway-infra(aa63dae6-4b4b-4878-abab-d854ecd6035f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f"
Jul 16 22:41:54.403058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:54.403035    2555 scope.go:117] "RemoveContainer" containerID="860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b"
Jul 16 22:41:54.403451 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:54.403217    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6_odh-ai-gateway-infra(aa63dae6-4b4b-4878-abab-d854ecd6035f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f"
Jul 16 22:41:54.670485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:54.670395    2555 scope.go:117] "RemoveContainer" containerID="48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d"
Jul 16 22:41:55.410417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:55.410383    2555 generic.go:356] "Generic (PLEG): container finished" podID="219302cc-e69d-4d11-9747-25e23696e572" containerID="fdb6ac32c78019e7e43f64c41072777aa12ffe70b1256c1d88b2dac766eddeda" exitCode=6
Jul 16 22:41:55.410866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:55.410474    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" event={"ID":"219302cc-e69d-4d11-9747-25e23696e572","Type":"ContainerDied","Data":"fdb6ac32c78019e7e43f64c41072777aa12ffe70b1256c1d88b2dac766eddeda"}
Jul 16 22:41:55.410866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:55.410537    2555 scope.go:117] "RemoveContainer" containerID="48426186b0762d4ead8db6213f659886432b56915416380f634d63545edfdb4d"
Jul 16 22:41:55.410866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:55.410813    2555 scope.go:117] "RemoveContainer" containerID="fdb6ac32c78019e7e43f64c41072777aa12ffe70b1256c1d88b2dac766eddeda"
Jul 16 22:41:55.411041 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:55.411019    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr_odh-ai-gateway-infra(219302cc-e69d-4d11-9747-25e23696e572)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" podUID="219302cc-e69d-4d11-9747-25e23696e572"
Jul 16 22:41:55.674608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:55.674503    2555 scope.go:117] "RemoveContainer" containerID="34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941"
Jul 16 22:41:56.419604 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.419574    2555 generic.go:356] "Generic (PLEG): container finished" podID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerID="4c6139dacfd745c82effdc79f9d10385e647f4b565baaa9332b3582037319a95" exitCode=6
Jul 16 22:41:56.420016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.419643    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" event={"ID":"7e5c1b8b-9173-4735-930e-d0c338fc3f70","Type":"ContainerDied","Data":"4c6139dacfd745c82effdc79f9d10385e647f4b565baaa9332b3582037319a95"}
Jul 16 22:41:56.420016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.419706    2555 scope.go:117] "RemoveContainer" containerID="34a3c269832dacc3b5e4a80187f40d1040a7247b0b7255c6b20043c852a55941"
Jul 16 22:41:56.420152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.420049    2555 scope.go:117] "RemoveContainer" containerID="4c6139dacfd745c82effdc79f9d10385e647f4b565baaa9332b3582037319a95"
Jul 16 22:41:56.420308 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:41:56.420287    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7_odh-ai-gateway-infra(7e5c1b8b-9173-4735-930e-d0c338fc3f70)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70"
Jul 16 22:41:56.447617 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.447590    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:56.591980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.591953    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:56.707954 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.707863    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca\") pod \"219302cc-e69d-4d11-9747-25e23696e572\" (UID: \"219302cc-e69d-4d11-9747-25e23696e572\") "
Jul 16 22:41:56.708264 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.708241    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "219302cc-e69d-4d11-9747-25e23696e572" (UID: "219302cc-e69d-4d11-9747-25e23696e572"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:56.808734 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:56.808702    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/219302cc-e69d-4d11-9747-25e23696e572-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:57.428198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.428158    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr" event={"ID":"219302cc-e69d-4d11-9747-25e23696e572","Type":"ContainerDied","Data":"20cdc143d8a41410a51d482c1b2385b808720fb8a0749debea559faaa6360571"}
Jul 16 22:41:57.428672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.428221    2555 scope.go:117] "RemoveContainer" containerID="fdb6ac32c78019e7e43f64c41072777aa12ffe70b1256c1d88b2dac766eddeda"
Jul 16 22:41:57.428672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.428174    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"
Jul 16 22:41:57.451143 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.451115    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:57.471991 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.471957    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:57.494518 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.494482    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-b7zcr"]
Jul 16 22:41:57.591603 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.591583    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:57.675836 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.675802    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="219302cc-e69d-4d11-9747-25e23696e572" path="/var/lib/kubelet/pods/219302cc-e69d-4d11-9747-25e23696e572/volumes"
Jul 16 22:41:57.716358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.716295    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca\") pod \"7e5c1b8b-9173-4735-930e-d0c338fc3f70\" (UID: \"7e5c1b8b-9173-4735-930e-d0c338fc3f70\") "
Jul 16 22:41:57.716636 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.716615    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "7e5c1b8b-9173-4735-930e-d0c338fc3f70" (UID: "7e5c1b8b-9173-4735-930e-d0c338fc3f70"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:41:57.817752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:57.817707    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e5c1b8b-9173-4735-930e-d0c338fc3f70-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:41:58.435573 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:58.435545    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"
Jul 16 22:41:58.435986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:58.435591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7" event={"ID":"7e5c1b8b-9173-4735-930e-d0c338fc3f70","Type":"ContainerDied","Data":"bfa3c4397ac6d1393ef1b9eb072e56312b902d1c9841cdc3d47c3b4251d75d28"}
Jul 16 22:41:58.435986 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:58.435649    2555 scope.go:117] "RemoveContainer" containerID="4c6139dacfd745c82effdc79f9d10385e647f4b565baaa9332b3582037319a95"
Jul 16 22:41:58.483154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:58.483118    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:58.506509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:58.506454    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-wkkp7"]
Jul 16 22:41:59.677098 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:41:59.677064    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" path="/var/lib/kubelet/pods/7e5c1b8b-9173-4735-930e-d0c338fc3f70/volumes"
Jul 16 22:42:00.381069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:00.381029    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:42:04.110536 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:04.110507    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:42:04.671012 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:04.670981    2555 scope.go:117] "RemoveContainer" containerID="860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b"
Jul 16 22:42:05.478606 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:05.478569    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerID="d227923ddff0e9a2ceecaf1f6fcbc3098b225cf1f8f34557ffaed13568e99dff" exitCode=6
Jul 16 22:42:05.479055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:05.478638    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" event={"ID":"aa63dae6-4b4b-4878-abab-d854ecd6035f","Type":"ContainerDied","Data":"d227923ddff0e9a2ceecaf1f6fcbc3098b225cf1f8f34557ffaed13568e99dff"}
Jul 16 22:42:05.479055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:05.478696    2555 scope.go:117] "RemoveContainer" containerID="860e30e2f14b328775b37b4af0da12aa33741fddc4e302b156bbbf015fca4f5b"
Jul 16 22:42:05.479055 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:05.479023    2555 scope.go:117] "RemoveContainer" containerID="d227923ddff0e9a2ceecaf1f6fcbc3098b225cf1f8f34557ffaed13568e99dff"
Jul 16 22:42:05.479305 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:05.479284    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6_odh-ai-gateway-infra(aa63dae6-4b4b-4878-abab-d854ecd6035f)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f"
Jul 16 22:42:06.531749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:06.531721    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:42:06.668265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:06.668241    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:42:06.798419 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:06.798383    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca\") pod \"aa63dae6-4b4b-4878-abab-d854ecd6035f\" (UID: \"aa63dae6-4b4b-4878-abab-d854ecd6035f\") "
Jul 16 22:42:06.798747 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:06.798725    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "aa63dae6-4b4b-4878-abab-d854ecd6035f" (UID: "aa63dae6-4b4b-4878-abab-d854ecd6035f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:06.899029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:06.898959    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aa63dae6-4b4b-4878-abab-d854ecd6035f-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:07.493086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.493051    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6" event={"ID":"aa63dae6-4b4b-4878-abab-d854ecd6035f","Type":"ContainerDied","Data":"4307a30194ea9900b56b240e505b6bc9c8bc99bc6877172098e8b96cbf4805f2"}
Jul 16 22:42:07.493286 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.493107    2555 scope.go:117] "RemoveContainer" containerID="d227923ddff0e9a2ceecaf1f6fcbc3098b225cf1f8f34557ffaed13568e99dff"
Jul 16 22:42:07.493286 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.493064    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"
Jul 16 22:42:07.518591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.518566    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:42:07.523824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.523803    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-c8ck6"]
Jul 16 22:42:07.676207 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:07.676176    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" path="/var/lib/kubelet/pods/aa63dae6-4b4b-4878-abab-d854ecd6035f/volumes"
Jul 16 22:42:15.401147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.401107    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" podUID="c29f292c-3571-45a1-a900-131051d02496" containerName="main" containerID="cri-o://1289c4e02d50d2043b24452558e621f0317552efb3447af2eb7c3a071bf42b63" gracePeriod=45
Jul 16 22:42:15.547608 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.547575    2555 generic.go:356] "Generic (PLEG): container finished" podID="c29f292c-3571-45a1-a900-131051d02496" containerID="1289c4e02d50d2043b24452558e621f0317552efb3447af2eb7c3a071bf42b63" exitCode=0
Jul 16 22:42:15.547795 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.547650    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" event={"ID":"c29f292c-3571-45a1-a900-131051d02496","Type":"ContainerDied","Data":"1289c4e02d50d2043b24452558e621f0317552efb3447af2eb7c3a071bf42b63"}
Jul 16 22:42:15.657965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.657906    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:42:15.670940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.670915    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.671086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.670972    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rcmjt\" (UniqueName: \"kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.671086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671006    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.671086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671033    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.671319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671190    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:15.671376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671318    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home" (OuterVolumeSpecName: "home") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:15.671376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671347    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-tmp-dir\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:15.671376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.671360    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache" (OuterVolumeSpecName: "model-cache") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:15.673266 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.673244    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt" (OuterVolumeSpecName: "kube-api-access-rcmjt") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "kube-api-access-rcmjt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:42:15.771808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.771779    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.771808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.771815    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs\") pod \"c29f292c-3571-45a1-a900-131051d02496\" (UID: \"c29f292c-3571-45a1-a900-131051d02496\") "
Jul 16 22:42:15.772032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.771963    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcmjt\" (UniqueName: \"kubernetes.io/projected/c29f292c-3571-45a1-a900-131051d02496-kube-api-access-rcmjt\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:15.772032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.771974    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-home\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:15.772032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.771983    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-model-cache\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:15.774003 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.773973    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:42:15.774120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.774026    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm" (OuterVolumeSpecName: "dshm") pod "c29f292c-3571-45a1-a900-131051d02496" (UID: "c29f292c-3571-45a1-a900-131051d02496"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:42:15.872560 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.872530    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c29f292c-3571-45a1-a900-131051d02496-dshm\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:15.872560 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:15.872555    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c29f292c-3571-45a1-a900-131051d02496-tls-certs\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:16.553829 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:16.553803    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"
Jul 16 22:42:16.554296 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:16.553844    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws" event={"ID":"c29f292c-3571-45a1-a900-131051d02496","Type":"ContainerDied","Data":"20d7b8912a47fafdcdff0e0e1c122636d5e45b2226fcf333423bdd0fea26e5b5"}
Jul 16 22:42:16.554296 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:16.553892    2555 scope.go:117] "RemoveContainer" containerID="1289c4e02d50d2043b24452558e621f0317552efb3447af2eb7c3a071bf42b63"
Jul 16 22:42:16.582820 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:16.582791    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:42:16.587150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:16.587125    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-dbc614/auth-test-model-dbc614-kserve-565b7bcfff-w46ws"]
Jul 16 22:42:17.676414 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:17.676386    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c29f292c-3571-45a1-a900-131051d02496" path="/var/lib/kubelet/pods/c29f292c-3571-45a1-a900-131051d02496/volumes"
Jul 16 22:42:20.893734 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:20.893695    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:42:21.529627 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529593    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:21.529932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529920    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529933    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529941    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529946    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529959    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529964    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529970    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.529982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529975    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529989    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.529994    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530004    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530009    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530014    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530018    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530027    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530032    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530037    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29f292c-3571-45a1-a900-131051d02496" containerName="main"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530042    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29f292c-3571-45a1-a900-131051d02496" containerName="main"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530093    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530099    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530104    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530110    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530116    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530121    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530128    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29f292c-3571-45a1-a900-131051d02496" containerName="main"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530135    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="219302cc-e69d-4d11-9747-25e23696e572" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530140    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:21.530183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.530144    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa63dae6-4b4b-4878-abab-d854ecd6035f" containerName="revoke-keys"
Jul 16 22:42:21.534751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.534734    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:21.538040 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.538005    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:42:21.538348 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.538325    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:42:21.541250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.541215    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:21.616189 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.616153    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5hc5x\" (UID: \"05264832-19e3-48a5-963c-c8db56b46657\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:21.717535 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.717500    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5hc5x\" (UID: \"05264832-19e3-48a5-963c-c8db56b46657\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:21.718069 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.718047    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-5hc5x\" (UID: \"05264832-19e3-48a5-963c-c8db56b46657\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:21.847126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.847033    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:21.984126 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:21.984093    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:21.986020 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:42:21.985991    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05264832_19e3_48a5_963c_c8db56b46657.slice/crio-10ca7262d1a68bf3b49f97d79796310830b06f29dbc7e67de920351184ff5b23 WatchSource:0}: Error finding container 10ca7262d1a68bf3b49f97d79796310830b06f29dbc7e67de920351184ff5b23: Status 404 returned error can't find the container with id 10ca7262d1a68bf3b49f97d79796310830b06f29dbc7e67de920351184ff5b23
Jul 16 22:42:22.588446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:22.588411    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerStarted","Data":"16cf4940dfcae571b7091de79fa3b4968bacfe269274cb110a0cd542c2601bc5"}
Jul 16 22:42:22.588649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:22.588486    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerStarted","Data":"10ca7262d1a68bf3b49f97d79796310830b06f29dbc7e67de920351184ff5b23"}
Jul 16 22:42:22.606525 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:22.606475    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" podStartSLOduration=1.6064441139999999 podStartE2EDuration="1.606444114s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:22.604688227 +0000 UTC m=+3197.441742027" watchObservedRunningTime="2026-07-16 22:42:22.606444114 +0000 UTC m=+3197.443497913"
Jul 16 22:42:23.595895 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:23.595861    2555 generic.go:356] "Generic (PLEG): container finished" podID="05264832-19e3-48a5-963c-c8db56b46657" containerID="16cf4940dfcae571b7091de79fa3b4968bacfe269274cb110a0cd542c2601bc5" exitCode=6
Jul 16 22:42:23.596367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:23.595936    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerDied","Data":"16cf4940dfcae571b7091de79fa3b4968bacfe269274cb110a0cd542c2601bc5"}
Jul 16 22:42:23.596367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:23.596147    2555 scope.go:117] "RemoveContainer" containerID="16cf4940dfcae571b7091de79fa3b4968bacfe269274cb110a0cd542c2601bc5"
Jul 16 22:42:24.603496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:24.603394    2555 generic.go:356] "Generic (PLEG): container finished" podID="05264832-19e3-48a5-963c-c8db56b46657" containerID="4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db" exitCode=6
Jul 16 22:42:24.603496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:24.603446    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerDied","Data":"4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db"}
Jul 16 22:42:24.603941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:24.603510    2555 scope.go:117] "RemoveContainer" containerID="16cf4940dfcae571b7091de79fa3b4968bacfe269274cb110a0cd542c2601bc5"
Jul 16 22:42:24.603941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:24.603689    2555 scope.go:117] "RemoveContainer" containerID="4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db"
Jul 16 22:42:24.604029 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:24.603936    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5hc5x_odh-ai-gateway-infra(05264832-19e3-48a5-963c-c8db56b46657)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" podUID="05264832-19e3-48a5-963c-c8db56b46657"
Jul 16 22:42:25.611107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:25.611077    2555 scope.go:117] "RemoveContainer" containerID="4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db"
Jul 16 22:42:25.611590 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:25.611317    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5hc5x_odh-ai-gateway-infra(05264832-19e3-48a5-963c-c8db56b46657)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" podUID="05264832-19e3-48a5-963c-c8db56b46657"
Jul 16 22:42:29.807937 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.807901    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:29.808375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.808267    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:29.808375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.808280    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e5c1b8b-9173-4735-930e-d0c338fc3f70" containerName="revoke-keys"
Jul 16 22:42:29.812689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.812668    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:29.820295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.819910    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:29.855850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.855814    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:29.859337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.859320    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:29.867943 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.867920    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:29.886562 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.886536    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49\" (UID: \"f78f9152-619d-42f5-abe2-634b714896ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:29.886693 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.886582    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv\" (UID: \"23ca0e14-333e-463f-b163-47cb896b58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:29.987743 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.987711    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49\" (UID: \"f78f9152-619d-42f5-abe2-634b714896ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:29.987929 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.987765    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv\" (UID: \"23ca0e14-333e-463f-b163-47cb896b58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:29.988356 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.988331    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49\" (UID: \"f78f9152-619d-42f5-abe2-634b714896ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:29.988457 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:29.988412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv\" (UID: \"23ca0e14-333e-463f-b163-47cb896b58a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:30.124591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.124505    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:30.171635 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.171595    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:30.283122 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.283088    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:30.288701 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:42:30.288671    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23ca0e14_333e_463f_b163_47cb896b58a7.slice/crio-33145b40bf4bba706e6729b752b67e3d00297c8733511b58a68cf7d1067685f2 WatchSource:0}: Error finding container 33145b40bf4bba706e6729b752b67e3d00297c8733511b58a68cf7d1067685f2: Status 404 returned error can't find the container with id 33145b40bf4bba706e6729b752b67e3d00297c8733511b58a68cf7d1067685f2
Jul 16 22:42:30.338850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.338825    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:30.340510 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:42:30.340479    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf78f9152_619d_42f5_abe2_634b714896ff.slice/crio-fbce666254413e31a1daa32da3577d51a9617f4f9122aefae697fdf4b1a5bd0b WatchSource:0}: Error finding container fbce666254413e31a1daa32da3577d51a9617f4f9122aefae697fdf4b1a5bd0b: Status 404 returned error can't find the container with id fbce666254413e31a1daa32da3577d51a9617f4f9122aefae697fdf4b1a5bd0b
Jul 16 22:42:30.639643 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.639611    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerStarted","Data":"24320d5cf09cb1c5bb1562ea47a5007184ff729eb82e065ea140f08a1b80b541"}
Jul 16 22:42:30.639783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.639681    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerStarted","Data":"fbce666254413e31a1daa32da3577d51a9617f4f9122aefae697fdf4b1a5bd0b"}
Jul 16 22:42:30.649003 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.648966    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerStarted","Data":"6d46aea3be99518bd5334af53827860ef45d84c4bdbdf787d8d57035adc4a312"}
Jul 16 22:42:30.649191 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.649062    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerStarted","Data":"33145b40bf4bba706e6729b752b67e3d00297c8733511b58a68cf7d1067685f2"}
Jul 16 22:42:30.663401 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.663357    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" podStartSLOduration=1.6633428270000001 podStartE2EDuration="1.663342827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:30.661384716 +0000 UTC m=+3205.498438516" watchObservedRunningTime="2026-07-16 22:42:30.663342827 +0000 UTC m=+3205.500396627"
Jul 16 22:42:30.684740 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:30.684697    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" podStartSLOduration=1.684683099 podStartE2EDuration="1.684683099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:30.683903236 +0000 UTC m=+3205.520957033" watchObservedRunningTime="2026-07-16 22:42:30.684683099 +0000 UTC m=+3205.521736898"
Jul 16 22:42:31.656214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.656175    2555 generic.go:356] "Generic (PLEG): container finished" podID="23ca0e14-333e-463f-b163-47cb896b58a7" containerID="6d46aea3be99518bd5334af53827860ef45d84c4bdbdf787d8d57035adc4a312" exitCode=6
Jul 16 22:42:31.656714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.656258    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerDied","Data":"6d46aea3be99518bd5334af53827860ef45d84c4bdbdf787d8d57035adc4a312"}
Jul 16 22:42:31.656714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.656449    2555 scope.go:117] "RemoveContainer" containerID="6d46aea3be99518bd5334af53827860ef45d84c4bdbdf787d8d57035adc4a312"
Jul 16 22:42:31.658043 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.658020    2555 generic.go:356] "Generic (PLEG): container finished" podID="f78f9152-619d-42f5-abe2-634b714896ff" containerID="24320d5cf09cb1c5bb1562ea47a5007184ff729eb82e065ea140f08a1b80b541" exitCode=6
Jul 16 22:42:31.658138 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.658078    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerDied","Data":"24320d5cf09cb1c5bb1562ea47a5007184ff729eb82e065ea140f08a1b80b541"}
Jul 16 22:42:31.658265 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:31.658253    2555 scope.go:117] "RemoveContainer" containerID="24320d5cf09cb1c5bb1562ea47a5007184ff729eb82e065ea140f08a1b80b541"
Jul 16 22:42:32.307553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.307520    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:42:32.665840 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.665811    2555 generic.go:356] "Generic (PLEG): container finished" podID="f78f9152-619d-42f5-abe2-634b714896ff" containerID="7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425" exitCode=6
Jul 16 22:42:32.666319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.665898    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerDied","Data":"7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425"}
Jul 16 22:42:32.666319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.665953    2555 scope.go:117] "RemoveContainer" containerID="24320d5cf09cb1c5bb1562ea47a5007184ff729eb82e065ea140f08a1b80b541"
Jul 16 22:42:32.666319 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.666124    2555 scope.go:117] "RemoveContainer" containerID="7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425"
Jul 16 22:42:32.666520 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:32.666351    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49_odh-ai-gateway-infra(f78f9152-619d-42f5-abe2-634b714896ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" podUID="f78f9152-619d-42f5-abe2-634b714896ff"
Jul 16 22:42:32.667922 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.667898    2555 generic.go:356] "Generic (PLEG): container finished" podID="23ca0e14-333e-463f-b163-47cb896b58a7" containerID="316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4" exitCode=6
Jul 16 22:42:32.668036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.667983    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerDied","Data":"316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4"}
Jul 16 22:42:32.668197 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.668185    2555 scope.go:117] "RemoveContainer" containerID="316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4"
Jul 16 22:42:32.668351 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:32.668336    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv_odh-ai-gateway-infra(23ca0e14-333e-463f-b163-47cb896b58a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" podUID="23ca0e14-333e-463f-b163-47cb896b58a7"
Jul 16 22:42:32.677866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:32.677852    2555 scope.go:117] "RemoveContainer" containerID="6d46aea3be99518bd5334af53827860ef45d84c4bdbdf787d8d57035adc4a312"
Jul 16 22:42:33.675876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:33.675853    2555 scope.go:117] "RemoveContainer" containerID="316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4"
Jul 16 22:42:33.676307 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:33.676069    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv_odh-ai-gateway-infra(23ca0e14-333e-463f-b163-47cb896b58a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" podUID="23ca0e14-333e-463f-b163-47cb896b58a7"
Jul 16 22:42:33.677253 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:33.677235    2555 scope.go:117] "RemoveContainer" containerID="7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425"
Jul 16 22:42:33.677414 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:33.677397    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49_odh-ai-gateway-infra(f78f9152-619d-42f5-abe2-634b714896ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" podUID="f78f9152-619d-42f5-abe2-634b714896ff"
Jul 16 22:42:38.671070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:38.671037    2555 scope.go:117] "RemoveContainer" containerID="4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db"
Jul 16 22:42:39.712262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:39.712161    2555 generic.go:356] "Generic (PLEG): container finished" podID="05264832-19e3-48a5-963c-c8db56b46657" containerID="037a3f37fffb7a45881601fe0e32e1c0801466655134f6f18919bca7f6f50f38" exitCode=6
Jul 16 22:42:39.712262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:39.712215    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerDied","Data":"037a3f37fffb7a45881601fe0e32e1c0801466655134f6f18919bca7f6f50f38"}
Jul 16 22:42:39.712262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:39.712262    2555 scope.go:117] "RemoveContainer" containerID="4bf772d9951e1eeab523b7f1abc830c4d4410e67cfbe34d5440e7ede23c248db"
Jul 16 22:42:39.712852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:39.712672    2555 scope.go:117] "RemoveContainer" containerID="037a3f37fffb7a45881601fe0e32e1c0801466655134f6f18919bca7f6f50f38"
Jul 16 22:42:39.712909 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:39.712872    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-5hc5x_odh-ai-gateway-infra(05264832-19e3-48a5-963c-c8db56b46657)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" podUID="05264832-19e3-48a5-963c-c8db56b46657"
Jul 16 22:42:40.743093 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.743058    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:40.881067 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.881046    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:40.960281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960250    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:42:40.960632 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960618    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.960689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960634    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.960689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960648    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.960689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960653    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.960787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960720    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.960787 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.960728    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:42:40.963642 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.963628    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:42:40.972899 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.972870    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:42:40.986132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.986102    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca\") pod \"05264832-19e3-48a5-963c-c8db56b46657\" (UID: \"05264832-19e3-48a5-963c-c8db56b46657\") "
Jul 16 22:42:40.986501 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:40.986450    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "05264832-19e3-48a5-963c-c8db56b46657" (UID: "05264832-19e3-48a5-963c-c8db56b46657"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:41.087698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.087665    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56\" (UID: \"1d757677-60df-43b2-92a2-b8842600b079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:42:41.087873 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.087795    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/05264832-19e3-48a5-963c-c8db56b46657-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:41.188169 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.188126    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56\" (UID: \"1d757677-60df-43b2-92a2-b8842600b079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:42:41.188790 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.188763    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56\" (UID: \"1d757677-60df-43b2-92a2-b8842600b079\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:42:41.274660 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.274624    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:42:41.411989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.411960    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:42:41.414062 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:42:41.414033    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d757677_60df_43b2_92a2_b8842600b079.slice/crio-195b25a535fdae006225e05dc4a16dbe023c73843aa40ec8048241a1351630ba WatchSource:0}: Error finding container 195b25a535fdae006225e05dc4a16dbe023c73843aa40ec8048241a1351630ba: Status 404 returned error can't find the container with id 195b25a535fdae006225e05dc4a16dbe023c73843aa40ec8048241a1351630ba
Jul 16 22:42:41.725597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.725574    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"
Jul 16 22:42:41.725730 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.725573    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x" event={"ID":"05264832-19e3-48a5-963c-c8db56b46657","Type":"ContainerDied","Data":"10ca7262d1a68bf3b49f97d79796310830b06f29dbc7e67de920351184ff5b23"}
Jul 16 22:42:41.725730 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.725706    2555 scope.go:117] "RemoveContainer" containerID="037a3f37fffb7a45881601fe0e32e1c0801466655134f6f18919bca7f6f50f38"
Jul 16 22:42:41.727218 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.727192    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerStarted","Data":"64f27d44bc6d781da1bad1304d2a625ae9973ac8207a7a2885258231719f3a03"}
Jul 16 22:42:41.727332 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.727242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerStarted","Data":"195b25a535fdae006225e05dc4a16dbe023c73843aa40ec8048241a1351630ba"}
Jul 16 22:42:41.755677 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.755623    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" podStartSLOduration=1.755604091 podStartE2EDuration="1.755604091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:42:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:42:41.754610776 +0000 UTC m=+3216.591664571" watchObservedRunningTime="2026-07-16 22:42:41.755604091 +0000 UTC m=+3216.592657892"
Jul 16 22:42:41.774989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.774953    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:41.777527 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:41.777506    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-5hc5x"]
Jul 16 22:42:42.734957 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:42.734922    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d757677-60df-43b2-92a2-b8842600b079" containerID="64f27d44bc6d781da1bad1304d2a625ae9973ac8207a7a2885258231719f3a03" exitCode=6
Jul 16 22:42:42.735102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:42.734994    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerDied","Data":"64f27d44bc6d781da1bad1304d2a625ae9973ac8207a7a2885258231719f3a03"}
Jul 16 22:42:42.735191 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:42.735174    2555 scope.go:117] "RemoveContainer" containerID="64f27d44bc6d781da1bad1304d2a625ae9973ac8207a7a2885258231719f3a03"
Jul 16 22:42:43.679036 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:43.678957    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="05264832-19e3-48a5-963c-c8db56b46657" path="/var/lib/kubelet/pods/05264832-19e3-48a5-963c-c8db56b46657/volumes"
Jul 16 22:42:43.742302 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:43.742266    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d757677-60df-43b2-92a2-b8842600b079" containerID="ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2" exitCode=6
Jul 16 22:42:43.742482 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:43.742357    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerDied","Data":"ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2"}
Jul 16 22:42:43.742482 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:43.742415    2555 scope.go:117] "RemoveContainer" containerID="64f27d44bc6d781da1bad1304d2a625ae9973ac8207a7a2885258231719f3a03"
Jul 16 22:42:43.742661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:43.742644    2555 scope.go:117] "RemoveContainer" containerID="ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2"
Jul 16 22:42:43.742907 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:43.742888    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56_odh-ai-gateway-infra(1d757677-60df-43b2-92a2-b8842600b079)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" podUID="1d757677-60df-43b2-92a2-b8842600b079"
Jul 16 22:42:44.670601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:44.670570    2555 scope.go:117] "RemoveContainer" containerID="7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425"
Jul 16 22:42:44.750133 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:44.750105    2555 scope.go:117] "RemoveContainer" containerID="ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2"
Jul 16 22:42:44.750564 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:44.750279    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56_odh-ai-gateway-infra(1d757677-60df-43b2-92a2-b8842600b079)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" podUID="1d757677-60df-43b2-92a2-b8842600b079"
Jul 16 22:42:45.757581 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:45.757497    2555 generic.go:356] "Generic (PLEG): container finished" podID="f78f9152-619d-42f5-abe2-634b714896ff" containerID="cd23cae28586357be4b9c41ef76fcbcb7ea25ca7ec3701477e3a6847a9d699d4" exitCode=6
Jul 16 22:42:45.757985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:45.757575    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerDied","Data":"cd23cae28586357be4b9c41ef76fcbcb7ea25ca7ec3701477e3a6847a9d699d4"}
Jul 16 22:42:45.757985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:45.757635    2555 scope.go:117] "RemoveContainer" containerID="7189cd0eb48378cc4c03b31833aebc0799ea435895cdf4ae54e59f17dce8a425"
Jul 16 22:42:45.757985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:45.757927    2555 scope.go:117] "RemoveContainer" containerID="cd23cae28586357be4b9c41ef76fcbcb7ea25ca7ec3701477e3a6847a9d699d4"
Jul 16 22:42:45.758158 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:45.758135    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49_odh-ai-gateway-infra(f78f9152-619d-42f5-abe2-634b714896ff)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" podUID="f78f9152-619d-42f5-abe2-634b714896ff"
Jul 16 22:42:46.828515 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:46.828452    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:46.968671 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:46.968642    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:47.040630 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.040596    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca\") pod \"f78f9152-619d-42f5-abe2-634b714896ff\" (UID: \"f78f9152-619d-42f5-abe2-634b714896ff\") "
Jul 16 22:42:47.040928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.040908    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f78f9152-619d-42f5-abe2-634b714896ff" (UID: "f78f9152-619d-42f5-abe2-634b714896ff"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:47.141331 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.141243    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f78f9152-619d-42f5-abe2-634b714896ff-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:47.771229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.771199    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"
Jul 16 22:42:47.771411 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.771232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49" event={"ID":"f78f9152-619d-42f5-abe2-634b714896ff","Type":"ContainerDied","Data":"fbce666254413e31a1daa32da3577d51a9617f4f9122aefae697fdf4b1a5bd0b"}
Jul 16 22:42:47.771411 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.771295    2555 scope.go:117] "RemoveContainer" containerID="cd23cae28586357be4b9c41ef76fcbcb7ea25ca7ec3701477e3a6847a9d699d4"
Jul 16 22:42:47.792470 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.792445    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:47.799658 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:47.799635    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-7rw49"]
Jul 16 22:42:48.670832 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:48.670801    2555 scope.go:117] "RemoveContainer" containerID="316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4"
Jul 16 22:42:49.676504 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:49.676407    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f78f9152-619d-42f5-abe2-634b714896ff" path="/var/lib/kubelet/pods/f78f9152-619d-42f5-abe2-634b714896ff/volumes"
Jul 16 22:42:49.785367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:49.785328    2555 generic.go:356] "Generic (PLEG): container finished" podID="23ca0e14-333e-463f-b163-47cb896b58a7" containerID="069a95ab26d84dd8b492be84725e1669f209d98cee3917b07053a72bcf9d5752" exitCode=6
Jul 16 22:42:49.785576 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:49.785380    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerDied","Data":"069a95ab26d84dd8b492be84725e1669f209d98cee3917b07053a72bcf9d5752"}
Jul 16 22:42:49.785576 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:49.785429    2555 scope.go:117] "RemoveContainer" containerID="316bdba5245a118faaa153829c897ad57ec5aa27e3804d3b043d3b03a283ade4"
Jul 16 22:42:49.785709 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:49.785697    2555 scope.go:117] "RemoveContainer" containerID="069a95ab26d84dd8b492be84725e1669f209d98cee3917b07053a72bcf9d5752"
Jul 16 22:42:49.785943 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:42:49.785921    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv_odh-ai-gateway-infra(23ca0e14-333e-463f-b163-47cb896b58a7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" podUID="23ca0e14-333e-463f-b163-47cb896b58a7"
Jul 16 22:42:50.828725 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:50.828685    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:50.981393 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:50.981368    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:51.075349 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.075312    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca\") pod \"23ca0e14-333e-463f-b163-47cb896b58a7\" (UID: \"23ca0e14-333e-463f-b163-47cb896b58a7\") "
Jul 16 22:42:51.075693 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.075671    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "23ca0e14-333e-463f-b163-47cb896b58a7" (UID: "23ca0e14-333e-463f-b163-47cb896b58a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:42:51.176083 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.175993    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/23ca0e14-333e-463f-b163-47cb896b58a7-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:42:51.799590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.799563    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"
Jul 16 22:42:51.799590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.799577    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv" event={"ID":"23ca0e14-333e-463f-b163-47cb896b58a7","Type":"ContainerDied","Data":"33145b40bf4bba706e6729b752b67e3d00297c8733511b58a68cf7d1067685f2"}
Jul 16 22:42:51.799870 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.799637    2555 scope.go:117] "RemoveContainer" containerID="069a95ab26d84dd8b492be84725e1669f209d98cee3917b07053a72bcf9d5752"
Jul 16 22:42:51.824454 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.824427    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:51.829051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:51.829028    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-7nvjv"]
Jul 16 22:42:53.677413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:53.677375    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" path="/var/lib/kubelet/pods/23ca0e14-333e-463f-b163-47cb896b58a7/volumes"
Jul 16 22:42:58.427136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:42:58.427056    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:43:00.670387 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:00.670345    2555 scope.go:117] "RemoveContainer" containerID="ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2"
Jul 16 22:43:01.857713 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:01.857677    2555 generic.go:356] "Generic (PLEG): container finished" podID="1d757677-60df-43b2-92a2-b8842600b079" containerID="0ce91e727cbbf57610c0ef0918ab99ff4ddd4a7fc02d964af843ba227805736e" exitCode=6
Jul 16 22:43:01.858149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:01.857751    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerDied","Data":"0ce91e727cbbf57610c0ef0918ab99ff4ddd4a7fc02d964af843ba227805736e"}
Jul 16 22:43:01.858149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:01.857813    2555 scope.go:117] "RemoveContainer" containerID="ad466ad004f3db854e04b1c4a32c9323e13213c6323c88cb1b22908e47a6e8c2"
Jul 16 22:43:01.858149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:01.858103    2555 scope.go:117] "RemoveContainer" containerID="0ce91e727cbbf57610c0ef0918ab99ff4ddd4a7fc02d964af843ba227805736e"
Jul 16 22:43:01.858350 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:01.858326    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56_odh-ai-gateway-infra(1d757677-60df-43b2-92a2-b8842600b079)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" podUID="1d757677-60df-43b2-92a2-b8842600b079"
Jul 16 22:43:02.891588 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:02.891564    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:43:03.027783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.027759    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:43:03.184019 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.183933    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca\") pod \"1d757677-60df-43b2-92a2-b8842600b079\" (UID: \"1d757677-60df-43b2-92a2-b8842600b079\") "
Jul 16 22:43:03.184321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.184299    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "1d757677-60df-43b2-92a2-b8842600b079" (UID: "1d757677-60df-43b2-92a2-b8842600b079"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:03.284526 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.284493    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1d757677-60df-43b2-92a2-b8842600b079-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:43:03.871201 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.871172    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"
Jul 16 22:43:03.871373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.871219    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56" event={"ID":"1d757677-60df-43b2-92a2-b8842600b079","Type":"ContainerDied","Data":"195b25a535fdae006225e05dc4a16dbe023c73843aa40ec8048241a1351630ba"}
Jul 16 22:43:03.871373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.871285    2555 scope.go:117] "RemoveContainer" containerID="0ce91e727cbbf57610c0ef0918ab99ff4ddd4a7fc02d964af843ba227805736e"
Jul 16 22:43:03.889339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.889314    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:43:03.895812 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:03.895789    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4qm56"]
Jul 16 22:43:05.677196 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:05.677162    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d757677-60df-43b2-92a2-b8842600b079" path="/var/lib/kubelet/pods/1d757677-60df-43b2-92a2-b8842600b079/volumes"
Jul 16 22:43:15.006356 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:15.006322    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:43:19.921930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.921896    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922241    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922253    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922261    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922267    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922277    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922283    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922289    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922294    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922301    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922306    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922312    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922324    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922329    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922334    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922339    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922350    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922355    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922363    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.922374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922367    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922419    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922427    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922433    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922441    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922447    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922451    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="05264832-19e3-48a5-963c-c8db56b46657" containerName="revoke-keys"
Jul 16 22:43:19.923010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.922457    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:19.925408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.925386    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:19.928395 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.928373    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:43:19.928518 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.928420    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:43:19.935237 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:19.935210    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:20.019597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.019556    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g\" (UID: \"55d55cac-5121-4978-8f66-0db1de244bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:20.120861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.120820    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g\" (UID: \"55d55cac-5121-4978-8f66-0db1de244bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:20.121452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.121433    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g\" (UID: \"55d55cac-5121-4978-8f66-0db1de244bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:20.236770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.236684    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:20.381553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.381530    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:20.383598 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:43:20.383570    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55d55cac_5121_4978_8f66_0db1de244bdd.slice/crio-de81da2996443e65336458a2111c976dc292c41daa9738e3461d321b8ae033ba WatchSource:0}: Error finding container de81da2996443e65336458a2111c976dc292c41daa9738e3461d321b8ae033ba: Status 404 returned error can't find the container with id de81da2996443e65336458a2111c976dc292c41daa9738e3461d321b8ae033ba
Jul 16 22:43:20.962415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.962328    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerStarted","Data":"732751bf5e27ae3576b3c1278079abefec093349b335f7eba536559c6333bc57"}
Jul 16 22:43:20.962415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.962386    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerStarted","Data":"de81da2996443e65336458a2111c976dc292c41daa9738e3461d321b8ae033ba"}
Jul 16 22:43:20.981303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:20.981255    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" podStartSLOduration=1.98124145 podStartE2EDuration="1.98124145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:20.979365591 +0000 UTC m=+3255.816419392" watchObservedRunningTime="2026-07-16 22:43:20.98124145 +0000 UTC m=+3255.818295250"
Jul 16 22:43:21.570908 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.570873    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:21.571337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.571323    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="23ca0e14-333e-463f-b163-47cb896b58a7" containerName="revoke-keys"
Jul 16 22:43:21.571401 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.571343    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d757677-60df-43b2-92a2-b8842600b079" containerName="revoke-keys"
Jul 16 22:43:21.571401 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.571351    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="f78f9152-619d-42f5-abe2-634b714896ff" containerName="revoke-keys"
Jul 16 22:43:21.574434 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.574414    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:21.583168 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.583138    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:21.631686 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.631655    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-fbw85\" (UID: \"d791f094-7180-4f4c-ba2a-bc0551270586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:21.732417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.732373    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-fbw85\" (UID: \"d791f094-7180-4f4c-ba2a-bc0551270586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:21.733020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.732998    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-fbw85\" (UID: \"d791f094-7180-4f4c-ba2a-bc0551270586\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:21.885591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.885524    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:21.969993 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.969959    2555 generic.go:356] "Generic (PLEG): container finished" podID="55d55cac-5121-4978-8f66-0db1de244bdd" containerID="732751bf5e27ae3576b3c1278079abefec093349b335f7eba536559c6333bc57" exitCode=6
Jul 16 22:43:21.970413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.969996    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerDied","Data":"732751bf5e27ae3576b3c1278079abefec093349b335f7eba536559c6333bc57"}
Jul 16 22:43:21.970413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:21.970253    2555 scope.go:117] "RemoveContainer" containerID="732751bf5e27ae3576b3c1278079abefec093349b335f7eba536559c6333bc57"
Jul 16 22:43:22.033637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.033606    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:22.035196 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:43:22.035172    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd791f094_7180_4f4c_ba2a_bc0551270586.slice/crio-7e4846aa2d7e79488255d7943a8a4f2bb425ca6444eb89920512bbc227fa09f7 WatchSource:0}: Error finding container 7e4846aa2d7e79488255d7943a8a4f2bb425ca6444eb89920512bbc227fa09f7: Status 404 returned error can't find the container with id 7e4846aa2d7e79488255d7943a8a4f2bb425ca6444eb89920512bbc227fa09f7
Jul 16 22:43:22.977696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.977656    2555 generic.go:356] "Generic (PLEG): container finished" podID="d791f094-7180-4f4c-ba2a-bc0551270586" containerID="a98df5843bc1b19a4b0600ffb65cc5c92250e6462b0d581fdca24620dc42cf58" exitCode=6
Jul 16 22:43:22.978106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.977745    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerDied","Data":"a98df5843bc1b19a4b0600ffb65cc5c92250e6462b0d581fdca24620dc42cf58"}
Jul 16 22:43:22.978106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.977799    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerStarted","Data":"7e4846aa2d7e79488255d7943a8a4f2bb425ca6444eb89920512bbc227fa09f7"}
Jul 16 22:43:22.978106 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.977901    2555 scope.go:117] "RemoveContainer" containerID="a98df5843bc1b19a4b0600ffb65cc5c92250e6462b0d581fdca24620dc42cf58"
Jul 16 22:43:22.979719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.979697    2555 generic.go:356] "Generic (PLEG): container finished" podID="55d55cac-5121-4978-8f66-0db1de244bdd" containerID="5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678" exitCode=6
Jul 16 22:43:22.979845 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.979740    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerDied","Data":"5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678"}
Jul 16 22:43:22.979845 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.979807    2555 scope.go:117] "RemoveContainer" containerID="732751bf5e27ae3576b3c1278079abefec093349b335f7eba536559c6333bc57"
Jul 16 22:43:22.979975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:22.979934    2555 scope.go:117] "RemoveContainer" containerID="5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678"
Jul 16 22:43:22.980112 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:22.980088    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g_odh-ai-gateway-infra(55d55cac-5121-4978-8f66-0db1de244bdd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" podUID="55d55cac-5121-4978-8f66-0db1de244bdd"
Jul 16 22:43:23.987680 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:23.987588    2555 generic.go:356] "Generic (PLEG): container finished" podID="d791f094-7180-4f4c-ba2a-bc0551270586" containerID="f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364" exitCode=6
Jul 16 22:43:23.988136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:23.987675    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerDied","Data":"f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364"}
Jul 16 22:43:23.988136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:23.987749    2555 scope.go:117] "RemoveContainer" containerID="a98df5843bc1b19a4b0600ffb65cc5c92250e6462b0d581fdca24620dc42cf58"
Jul 16 22:43:23.988136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:23.987844    2555 scope.go:117] "RemoveContainer" containerID="f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364"
Jul 16 22:43:23.988136 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:23.988077    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-fbw85_odh-ai-gateway-infra(d791f094-7180-4f4c-ba2a-bc0551270586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" podUID="d791f094-7180-4f4c-ba2a-bc0551270586"
Jul 16 22:43:23.989569 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:23.989554    2555 scope.go:117] "RemoveContainer" containerID="5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678"
Jul 16 22:43:23.989748 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:23.989730    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g_odh-ai-gateway-infra(55d55cac-5121-4978-8f66-0db1de244bdd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" podUID="55d55cac-5121-4978-8f66-0db1de244bdd"
Jul 16 22:43:24.996439 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:24.996412    2555 scope.go:117] "RemoveContainer" containerID="f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364"
Jul 16 22:43:24.996838 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:24.996643    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-fbw85_odh-ai-gateway-infra(d791f094-7180-4f4c-ba2a-bc0551270586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" podUID="d791f094-7180-4f4c-ba2a-bc0551270586"
Jul 16 22:43:29.882956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.882919    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:29.887472 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.887443    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:29.893821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.893788    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4\" (UID: \"2d421927-bf4f-4b56-87f3-6a4cc4de24dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:29.894113 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.894087    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:29.994654 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.994615    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4\" (UID: \"2d421927-bf4f-4b56-87f3-6a4cc4de24dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:29.995224 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:29.995205    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4\" (UID: \"2d421927-bf4f-4b56-87f3-6a4cc4de24dd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:30.198991 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.198879    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:30.343794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.343770    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:30.344506 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:43:30.344453    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d421927_bf4f_4b56_87f3_6a4cc4de24dd.slice/crio-ff8549a088b10a9b5bdc1b4ebe1d0954b42c00ee2f15d3179bf2553ec6a3a2f7 WatchSource:0}: Error finding container ff8549a088b10a9b5bdc1b4ebe1d0954b42c00ee2f15d3179bf2553ec6a3a2f7: Status 404 returned error can't find the container with id ff8549a088b10a9b5bdc1b4ebe1d0954b42c00ee2f15d3179bf2553ec6a3a2f7
Jul 16 22:43:30.499650 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.499574    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:43:30.504282 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.504260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.508513 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.508489    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bb55e2\"/\"rate-test-model-bb55e2-kserve-self-signed-certs\""
Jul 16 22:43:30.508654 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.508540    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bb55e2\"/\"kube-root-ca.crt\""
Jul 16 22:43:30.508719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.508653    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-bb55e2\"/\"default-dockercfg-hdb74\""
Jul 16 22:43:30.508770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.508718    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-bb55e2\"/\"openshift-service-ca.crt\""
Jul 16 22:43:30.515979 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.515959    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:43:30.601134 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601108    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.601302 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601140    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.601302 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601198    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7pfx9\" (UniqueName: \"kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.601425 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601308    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.601425 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601356    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.601538 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.601444    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.702719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702681    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.702891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702730    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.702891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702747    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.702891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702827    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7pfx9\" (UniqueName: \"kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.702891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702875    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.703112 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.702908    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.703112 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.703097    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.703252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.703227    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.703308 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.703243    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.705113 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.705088    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.705375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.705356    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.711691 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.711669    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7pfx9\" (UniqueName: \"kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9\") pod \"rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") " pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.827845 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.827812    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:30.969934 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:30.969911    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:43:30.974401 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:43:30.974370    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podaa829904_3666_434c_bb9b_b89e69bfaf42.slice/crio-f77c394b0bbc0400279f495d3c54bd1f18b936501d14e7478a090c5abfc3ee19 WatchSource:0}: Error finding container f77c394b0bbc0400279f495d3c54bd1f18b936501d14e7478a090c5abfc3ee19: Status 404 returned error can't find the container with id f77c394b0bbc0400279f495d3c54bd1f18b936501d14e7478a090c5abfc3ee19
Jul 16 22:43:31.031223 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.031197    2555 generic.go:356] "Generic (PLEG): container finished" podID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerID="feb7c313f45b82bcc512f7983cf48d83dd8d7c29b001f1162c320367fb06ade6" exitCode=6
Jul 16 22:43:31.031341 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.031280    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerDied","Data":"feb7c313f45b82bcc512f7983cf48d83dd8d7c29b001f1162c320367fb06ade6"}
Jul 16 22:43:31.031415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.031341    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerStarted","Data":"ff8549a088b10a9b5bdc1b4ebe1d0954b42c00ee2f15d3179bf2553ec6a3a2f7"}
Jul 16 22:43:31.031523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.031477    2555 scope.go:117] "RemoveContainer" containerID="feb7c313f45b82bcc512f7983cf48d83dd8d7c29b001f1162c320367fb06ade6"
Jul 16 22:43:31.033051 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.033030    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" event={"ID":"aa829904-3666-434c-bb9b-b89e69bfaf42","Type":"ContainerStarted","Data":"f77c394b0bbc0400279f495d3c54bd1f18b936501d14e7478a090c5abfc3ee19"}
Jul 16 22:43:31.072624 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:31.072194    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" podStartSLOduration=1.072171453 podStartE2EDuration="1.072171453s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:31.068870413 +0000 UTC m=+3265.905924213" watchObservedRunningTime="2026-07-16 22:43:31.072171453 +0000 UTC m=+3265.909225257"
Jul 16 22:43:32.040864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:32.040827    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" event={"ID":"aa829904-3666-434c-bb9b-b89e69bfaf42","Type":"ContainerStarted","Data":"c2bed011ffe38f988b15ac3d5987ec089d8799388dc5967112e8b7bf91794091"}
Jul 16 22:43:32.042591 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:32.042565    2555 generic.go:356] "Generic (PLEG): container finished" podID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerID="301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62" exitCode=6
Jul 16 22:43:32.042704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:32.042644    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerDied","Data":"301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62"}
Jul 16 22:43:32.042704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:32.042699    2555 scope.go:117] "RemoveContainer" containerID="feb7c313f45b82bcc512f7983cf48d83dd8d7c29b001f1162c320367fb06ade6"
Jul 16 22:43:32.042824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:32.042806    2555 scope.go:117] "RemoveContainer" containerID="301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62"
Jul 16 22:43:32.043016 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:32.042993    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4_odh-ai-gateway-infra(2d421927-bf4f-4b56-87f3-6a4cc4de24dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd"
Jul 16 22:43:33.056481 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:33.056427    2555 scope.go:117] "RemoveContainer" containerID="301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62"
Jul 16 22:43:33.056878 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:33.056628    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4_odh-ai-gateway-infra(2d421927-bf4f-4b56-87f3-6a4cc4de24dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd"
Jul 16 22:43:34.323170 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:34.323123    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:43:35.680150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:35.680111    2555 scope.go:117] "RemoveContainer" containerID="f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364"
Jul 16 22:43:36.077699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:36.077662    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerStarted","Data":"17377cf65d5fdc5b861231199e0574d8226aeda3917d8f635f5860e8f03da089"}
Jul 16 22:43:36.096640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:36.095945    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" podStartSLOduration=15.095924728 podStartE2EDuration="15.095924728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:36.095387114 +0000 UTC m=+3270.932440912" watchObservedRunningTime="2026-07-16 22:43:36.095924728 +0000 UTC m=+3270.932978530"
Jul 16 22:43:37.085473 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:37.085426    2555 generic.go:356] "Generic (PLEG): container finished" podID="d791f094-7180-4f4c-ba2a-bc0551270586" containerID="17377cf65d5fdc5b861231199e0574d8226aeda3917d8f635f5860e8f03da089" exitCode=6
Jul 16 22:43:37.085869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:37.085503    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerDied","Data":"17377cf65d5fdc5b861231199e0574d8226aeda3917d8f635f5860e8f03da089"}
Jul 16 22:43:37.085869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:37.085568    2555 scope.go:117] "RemoveContainer" containerID="f92ab420531de845d2cb2ab627a47bbfe98b271530c98470a155f74be7970364"
Jul 16 22:43:37.085965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:37.085882    2555 scope.go:117] "RemoveContainer" containerID="17377cf65d5fdc5b861231199e0574d8226aeda3917d8f635f5860e8f03da089"
Jul 16 22:43:37.086083 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:37.086064    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-fbw85_odh-ai-gateway-infra(d791f094-7180-4f4c-ba2a-bc0551270586)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" podUID="d791f094-7180-4f4c-ba2a-bc0551270586"
Jul 16 22:43:37.115011 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:37.114983    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:38.227033 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:38.227005    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:38.265020 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:38.264988    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca\") pod \"d791f094-7180-4f4c-ba2a-bc0551270586\" (UID: \"d791f094-7180-4f4c-ba2a-bc0551270586\") "
Jul 16 22:43:38.265417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:38.265391    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d791f094-7180-4f4c-ba2a-bc0551270586" (UID: "d791f094-7180-4f4c-ba2a-bc0551270586"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:38.366250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:38.366159    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d791f094-7180-4f4c-ba2a-bc0551270586-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:43:39.099893 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.099867    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"
Jul 16 22:43:39.100080 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.099867    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85" event={"ID":"d791f094-7180-4f4c-ba2a-bc0551270586","Type":"ContainerDied","Data":"7e4846aa2d7e79488255d7943a8a4f2bb425ca6444eb89920512bbc227fa09f7"}
Jul 16 22:43:39.100080 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.099996    2555 scope.go:117] "RemoveContainer" containerID="17377cf65d5fdc5b861231199e0574d8226aeda3917d8f635f5860e8f03da089"
Jul 16 22:43:39.132476 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.132423    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:39.137094 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.137070    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-fbw85"]
Jul 16 22:43:39.678102 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.678054    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" path="/var/lib/kubelet/pods/d791f094-7180-4f4c-ba2a-bc0551270586/volumes"
Jul 16 22:43:39.678857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:39.678838    2555 scope.go:117] "RemoveContainer" containerID="5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678"
Jul 16 22:43:40.109058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:40.109027    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerStarted","Data":"430bd18f293e358ca28508e98b9b5e019210af68cb0a4c108a465fc19103485a"}
Jul 16 22:43:40.828412 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:40.828374    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:40.828829 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:40.828429    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:40.841290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:40.841255    2555 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:41.009362 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009327    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:41.009768 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009754    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009770    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009810    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009820    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009884    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009967 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009894    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.009967 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.009899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:43:41.012891 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.012871    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:41.023233 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.023211    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:41.088965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.088856    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp\" (UID: \"00365eef-3b7b-4197-80b9-46cd44bc6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:41.116260 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.116232    2555 generic.go:356] "Generic (PLEG): container finished" podID="55d55cac-5121-4978-8f66-0db1de244bdd" containerID="430bd18f293e358ca28508e98b9b5e019210af68cb0a4c108a465fc19103485a" exitCode=6
Jul 16 22:43:41.116392 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.116315    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerDied","Data":"430bd18f293e358ca28508e98b9b5e019210af68cb0a4c108a465fc19103485a"}
Jul 16 22:43:41.116392 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.116387    2555 scope.go:117] "RemoveContainer" containerID="5c3a6f26365dce02a0030a9bdc9bd97d42b1e9df35543338046d988c3f231678"
Jul 16 22:43:41.116702 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.116664    2555 scope.go:117] "RemoveContainer" containerID="430bd18f293e358ca28508e98b9b5e019210af68cb0a4c108a465fc19103485a"
Jul 16 22:43:41.116933 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:41.116907    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g_odh-ai-gateway-infra(55d55cac-5121-4978-8f66-0db1de244bdd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" podUID="55d55cac-5121-4978-8f66-0db1de244bdd"
Jul 16 22:43:41.128231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.128210    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:43:41.141729 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.141694    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:41.189948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.189909    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp\" (UID: \"00365eef-3b7b-4197-80b9-46cd44bc6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:41.190611 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.190589    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp\" (UID: \"00365eef-3b7b-4197-80b9-46cd44bc6298\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:41.323722 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.323688    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:41.460653 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:41.460630    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:41.463058 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:43:41.463027    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00365eef_3b7b_4197_80b9_46cd44bc6298.slice/crio-f07dd739df37c88e84387e1a579b57b42c76b1ae110f68efbf74cffa48961a42 WatchSource:0}: Error finding container f07dd739df37c88e84387e1a579b57b42c76b1ae110f68efbf74cffa48961a42: Status 404 returned error can't find the container with id f07dd739df37c88e84387e1a579b57b42c76b1ae110f68efbf74cffa48961a42
Jul 16 22:43:42.125154 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.125125    2555 generic.go:356] "Generic (PLEG): container finished" podID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerID="48c96c27253a94bd049d38165e9f35b40b10796efd5f05cdfacbd414ee86e689" exitCode=6
Jul 16 22:43:42.125567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.125216    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerDied","Data":"48c96c27253a94bd049d38165e9f35b40b10796efd5f05cdfacbd414ee86e689"}
Jul 16 22:43:42.125567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.125265    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerStarted","Data":"f07dd739df37c88e84387e1a579b57b42c76b1ae110f68efbf74cffa48961a42"}
Jul 16 22:43:42.125567 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.125482    2555 scope.go:117] "RemoveContainer" containerID="48c96c27253a94bd049d38165e9f35b40b10796efd5f05cdfacbd414ee86e689"
Jul 16 22:43:42.247996 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.247970    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:42.300077 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.300046    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca\") pod \"55d55cac-5121-4978-8f66-0db1de244bdd\" (UID: \"55d55cac-5121-4978-8f66-0db1de244bdd\") "
Jul 16 22:43:42.300415 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.300389    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "55d55cac-5121-4978-8f66-0db1de244bdd" (UID: "55d55cac-5121-4978-8f66-0db1de244bdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:42.400927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:42.400899    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/55d55cac-5121-4978-8f66-0db1de244bdd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:43:43.132785 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.132696    2555 generic.go:356] "Generic (PLEG): container finished" podID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerID="09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda" exitCode=6
Jul 16 22:43:43.133227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.132776    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerDied","Data":"09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda"}
Jul 16 22:43:43.133227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.132836    2555 scope.go:117] "RemoveContainer" containerID="48c96c27253a94bd049d38165e9f35b40b10796efd5f05cdfacbd414ee86e689"
Jul 16 22:43:43.133227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.133014    2555 scope.go:117] "RemoveContainer" containerID="09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda"
Jul 16 22:43:43.133367 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:43.133242    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp_odh-ai-gateway-infra(00365eef-3b7b-4197-80b9-46cd44bc6298)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298"
Jul 16 22:43:43.134282 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.134262    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g" event={"ID":"55d55cac-5121-4978-8f66-0db1de244bdd","Type":"ContainerDied","Data":"de81da2996443e65336458a2111c976dc292c41daa9738e3461d321b8ae033ba"}
Jul 16 22:43:43.134355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.134279    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"
Jul 16 22:43:43.144955 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.144935    2555 scope.go:117] "RemoveContainer" containerID="430bd18f293e358ca28508e98b9b5e019210af68cb0a4c108a465fc19103485a"
Jul 16 22:43:43.172098 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.172066    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:43.174514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.174497    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-5wp2g"]
Jul 16 22:43:43.676281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:43.676244    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" path="/var/lib/kubelet/pods/55d55cac-5121-4978-8f66-0db1de244bdd/volumes"
Jul 16 22:43:44.141749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:44.141724    2555 scope.go:117] "RemoveContainer" containerID="09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda"
Jul 16 22:43:44.142209 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:44.141926    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp_odh-ai-gateway-infra(00365eef-3b7b-4197-80b9-46cd44bc6298)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298"
Jul 16 22:43:46.670845 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:46.670814    2555 scope.go:117] "RemoveContainer" containerID="301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62"
Jul 16 22:43:47.160886 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:47.160849    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerStarted","Data":"6e1dc681a3c2d8012e08d41deca53dfb4a5e2422c8d2924f1a348c6da8338a94"}
Jul 16 22:43:47.192484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:47.192420    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" podStartSLOduration=18.192403713 podStartE2EDuration="18.192403713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:47.191504781 +0000 UTC m=+3282.028558581" watchObservedRunningTime="2026-07-16 22:43:47.192403713 +0000 UTC m=+3282.029457512"
Jul 16 22:43:48.168532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:48.168490    2555 generic.go:356] "Generic (PLEG): container finished" podID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerID="6e1dc681a3c2d8012e08d41deca53dfb4a5e2422c8d2924f1a348c6da8338a94" exitCode=6
Jul 16 22:43:48.168956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:48.168500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerDied","Data":"6e1dc681a3c2d8012e08d41deca53dfb4a5e2422c8d2924f1a348c6da8338a94"}
Jul 16 22:43:48.168956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:48.168639    2555 scope.go:117] "RemoveContainer" containerID="301f5790e03015813aa2f1c46b6acf957fb46d428ef98026ffa82cdf22990c62"
Jul 16 22:43:48.168956 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:48.168930    2555 scope.go:117] "RemoveContainer" containerID="6e1dc681a3c2d8012e08d41deca53dfb4a5e2422c8d2924f1a348c6da8338a94"
Jul 16 22:43:48.169142 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:48.169118    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4_odh-ai-gateway-infra(2d421927-bf4f-4b56-87f3-6a4cc4de24dd)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd"
Jul 16 22:43:48.217808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:48.217769    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:49.311306 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:49.311284    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:49.365212 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:49.365186    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca\") pod \"2d421927-bf4f-4b56-87f3-6a4cc4de24dd\" (UID: \"2d421927-bf4f-4b56-87f3-6a4cc4de24dd\") "
Jul 16 22:43:49.365586 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:49.365564    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2d421927-bf4f-4b56-87f3-6a4cc4de24dd" (UID: "2d421927-bf4f-4b56-87f3-6a4cc4de24dd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:49.466750 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:49.466661    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2d421927-bf4f-4b56-87f3-6a4cc4de24dd-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:43:50.184215 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:50.184186    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"
Jul 16 22:43:50.184417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:50.184212    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4" event={"ID":"2d421927-bf4f-4b56-87f3-6a4cc4de24dd","Type":"ContainerDied","Data":"ff8549a088b10a9b5bdc1b4ebe1d0954b42c00ee2f15d3179bf2553ec6a3a2f7"}
Jul 16 22:43:50.184417 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:50.184286    2555 scope.go:117] "RemoveContainer" containerID="6e1dc681a3c2d8012e08d41deca53dfb4a5e2422c8d2924f1a348c6da8338a94"
Jul 16 22:43:50.206002 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:50.205947    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:50.208371 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:50.208349    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-dv5m4"]
Jul 16 22:43:51.676017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:51.675987    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" path="/var/lib/kubelet/pods/2d421927-bf4f-4b56-87f3-6a4cc4de24dd/volumes"
Jul 16 22:43:54.671479 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:54.671430    2555 scope.go:117] "RemoveContainer" containerID="09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda"
Jul 16 22:43:55.215070 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:55.215036    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerStarted","Data":"5c3f3d25279a634207f64aa5a71ffd2aab8058508b10bc0cfb9b438455cffd1b"}
Jul 16 22:43:55.233408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:55.233358    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" podStartSLOduration=15.233343982 podStartE2EDuration="15.233343982s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:43:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:43:55.231601623 +0000 UTC m=+3290.068655422" watchObservedRunningTime="2026-07-16 22:43:55.233343982 +0000 UTC m=+3290.070397781"
Jul 16 22:43:56.223183 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:56.223149    2555 generic.go:356] "Generic (PLEG): container finished" podID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerID="5c3f3d25279a634207f64aa5a71ffd2aab8058508b10bc0cfb9b438455cffd1b" exitCode=6
Jul 16 22:43:56.223597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:56.223226    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerDied","Data":"5c3f3d25279a634207f64aa5a71ffd2aab8058508b10bc0cfb9b438455cffd1b"}
Jul 16 22:43:56.223597 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:56.223297    2555 scope.go:117] "RemoveContainer" containerID="09ce1cc585495556a8d59b905d0b56c2d0d13140382dac05e4b5840fcaf4fbda"
Jul 16 22:43:56.223753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:56.223639    2555 scope.go:117] "RemoveContainer" containerID="5c3f3d25279a634207f64aa5a71ffd2aab8058508b10bc0cfb9b438455cffd1b"
Jul 16 22:43:56.223914 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:43:56.223890    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp_odh-ai-gateway-infra(00365eef-3b7b-4197-80b9-46cd44bc6298)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298"
Jul 16 22:43:56.244363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:56.244342    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:57.369283 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:57.369258    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:57.439553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:57.439517    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca\") pod \"00365eef-3b7b-4197-80b9-46cd44bc6298\" (UID: \"00365eef-3b7b-4197-80b9-46cd44bc6298\") "
Jul 16 22:43:57.439887 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:57.439865    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "00365eef-3b7b-4197-80b9-46cd44bc6298" (UID: "00365eef-3b7b-4197-80b9-46cd44bc6298"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:43:57.540224 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:57.540196    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/00365eef-3b7b-4197-80b9-46cd44bc6298-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:43:58.237534 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:58.237503    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"
Jul 16 22:43:58.237728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:58.237504    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp" event={"ID":"00365eef-3b7b-4197-80b9-46cd44bc6298","Type":"ContainerDied","Data":"f07dd739df37c88e84387e1a579b57b42c76b1ae110f68efbf74cffa48961a42"}
Jul 16 22:43:58.237728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:58.237635    2555 scope.go:117] "RemoveContainer" containerID="5c3f3d25279a634207f64aa5a71ffd2aab8058508b10bc0cfb9b438455cffd1b"
Jul 16 22:43:58.272438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:58.272405    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:58.279596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:58.279575    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-4zqvp"]
Jul 16 22:43:59.677032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:43:59.676994    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" path="/var/lib/kubelet/pods/00365eef-3b7b-4197-80b9-46cd44bc6298/volumes"
Jul 16 22:44:11.655853 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.655815    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:11.656413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656381    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.656413 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656402    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656415    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656424    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656445    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656454    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656495    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656504    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656515    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656523    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656532    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656540    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656557    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656567    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d791f094-7180-4f4c-ba2a-bc0551270586" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656587    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.656600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656596    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656694    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656708    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656717    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656725    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656732    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656740    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656748    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656757    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:11.657303 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.656765    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="00365eef-3b7b-4197-80b9-46cd44bc6298" containerName="revoke-keys"
Jul 16 22:44:11.661881 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.661858    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:11.667330 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.667301    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:44:11.667604 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.667586    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:44:11.679136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.679104    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:11.767687 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.767646    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-s897r\" (UID: \"bd3c465e-a15c-4f2c-9afc-49a7d306bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:11.869065 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.869010    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-s897r\" (UID: \"bd3c465e-a15c-4f2c-9afc-49a7d306bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:11.869679 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.869660    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-s897r\" (UID: \"bd3c465e-a15c-4f2c-9afc-49a7d306bc92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:11.972945 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:11.972846    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:12.129590 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:12.129565    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:12.131242 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:44:12.131209    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd3c465e_a15c_4f2c_9afc_49a7d306bc92.slice/crio-7e6d35a8130c3e6dfc54bd17a6fb8ffa81434256faf5d98cd548b71c69fded53 WatchSource:0}: Error finding container 7e6d35a8130c3e6dfc54bd17a6fb8ffa81434256faf5d98cd548b71c69fded53: Status 404 returned error can't find the container with id 7e6d35a8130c3e6dfc54bd17a6fb8ffa81434256faf5d98cd548b71c69fded53
Jul 16 22:44:12.316637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:12.316604    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" event={"ID":"bd3c465e-a15c-4f2c-9afc-49a7d306bc92","Type":"ContainerStarted","Data":"7e6d35a8130c3e6dfc54bd17a6fb8ffa81434256faf5d98cd548b71c69fded53"}
Jul 16 22:44:13.324559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:13.324520    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerID="0093446f9afa8322e3ae13f32a0e52063ff0e4d7f9225d9034719b6cc203b0d0" exitCode=6
Jul 16 22:44:13.324947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:13.324608    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" event={"ID":"bd3c465e-a15c-4f2c-9afc-49a7d306bc92","Type":"ContainerDied","Data":"0093446f9afa8322e3ae13f32a0e52063ff0e4d7f9225d9034719b6cc203b0d0"}
Jul 16 22:44:13.324947 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:13.324790    2555 scope.go:117] "RemoveContainer" containerID="0093446f9afa8322e3ae13f32a0e52063ff0e4d7f9225d9034719b6cc203b0d0"
Jul 16 22:44:14.331257 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:14.331221    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerID="28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7" exitCode=6
Jul 16 22:44:14.331763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:14.331299    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" event={"ID":"bd3c465e-a15c-4f2c-9afc-49a7d306bc92","Type":"ContainerDied","Data":"28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7"}
Jul 16 22:44:14.331763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:14.331353    2555 scope.go:117] "RemoveContainer" containerID="0093446f9afa8322e3ae13f32a0e52063ff0e4d7f9225d9034719b6cc203b0d0"
Jul 16 22:44:14.331763 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:14.331585    2555 scope.go:117] "RemoveContainer" containerID="28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7"
Jul 16 22:44:14.331933 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:14.331825    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-s897r_odh-ai-gateway-infra(bd3c465e-a15c-4f2c-9afc-49a7d306bc92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92"
Jul 16 22:44:15.338238 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:15.338201    2555 scope.go:117] "RemoveContainer" containerID="28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7"
Jul 16 22:44:15.338656 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:15.338435    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-s897r_odh-ai-gateway-infra(bd3c465e-a15c-4f2c-9afc-49a7d306bc92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92"
Jul 16 22:44:20.012771 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.012737    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:20.013165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.013084    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:20.013165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.013094    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d55cac-5121-4978-8f66-0db1de244bdd" containerName="revoke-keys"
Jul 16 22:44:20.013165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.013114    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:20.013165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.013119    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d421927-bf4f-4b56-87f3-6a4cc4de24dd" containerName="revoke-keys"
Jul 16 22:44:20.016159 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.016143    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:20.031419 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.031388    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:20.040433 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.040405    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r\" (UID: \"d5b32778-ae14-4b6a-9fd2-2565d14aa360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:20.141331 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.141293    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r\" (UID: \"d5b32778-ae14-4b6a-9fd2-2565d14aa360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:20.141944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.141926    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r\" (UID: \"d5b32778-ae14-4b6a-9fd2-2565d14aa360\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:20.326712 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.326685    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:20.477941 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:20.477914    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:20.479797 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:44:20.479772    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5b32778_ae14_4b6a_9fd2_2565d14aa360.slice/crio-df1dd733af9d452b1ba542b56369d49e81ed618f6b9f666e20d90f254c07febe WatchSource:0}: Error finding container df1dd733af9d452b1ba542b56369d49e81ed618f6b9f666e20d90f254c07febe: Status 404 returned error can't find the container with id df1dd733af9d452b1ba542b56369d49e81ed618f6b9f666e20d90f254c07febe
Jul 16 22:44:21.374030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:21.373992    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerID="512c78cd20adf96a9e2fb3d9c2501fe3401eeb0ca6b9103d93432b69c239e100" exitCode=6
Jul 16 22:44:21.374446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:21.374080    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" event={"ID":"d5b32778-ae14-4b6a-9fd2-2565d14aa360","Type":"ContainerDied","Data":"512c78cd20adf96a9e2fb3d9c2501fe3401eeb0ca6b9103d93432b69c239e100"}
Jul 16 22:44:21.374446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:21.374182    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" event={"ID":"d5b32778-ae14-4b6a-9fd2-2565d14aa360","Type":"ContainerStarted","Data":"df1dd733af9d452b1ba542b56369d49e81ed618f6b9f666e20d90f254c07febe"}
Jul 16 22:44:21.374446 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:21.374278    2555 scope.go:117] "RemoveContainer" containerID="512c78cd20adf96a9e2fb3d9c2501fe3401eeb0ca6b9103d93432b69c239e100"
Jul 16 22:44:22.381444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:22.381411    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerID="0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61" exitCode=6
Jul 16 22:44:22.381937 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:22.381500    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" event={"ID":"d5b32778-ae14-4b6a-9fd2-2565d14aa360","Type":"ContainerDied","Data":"0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61"}
Jul 16 22:44:22.381937 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:22.381568    2555 scope.go:117] "RemoveContainer" containerID="512c78cd20adf96a9e2fb3d9c2501fe3401eeb0ca6b9103d93432b69c239e100"
Jul 16 22:44:22.381937 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:22.381792    2555 scope.go:117] "RemoveContainer" containerID="0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61"
Jul 16 22:44:22.382074 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:22.382018    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r_odh-ai-gateway-infra(d5b32778-ae14-4b6a-9fd2-2565d14aa360)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360"
Jul 16 22:44:23.365751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:23.365719    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:44:23.389531 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:23.389509    2555 scope.go:117] "RemoveContainer" containerID="0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61"
Jul 16 22:44:23.389877 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:23.389694    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r_odh-ai-gateway-infra(d5b32778-ae14-4b6a-9fd2-2565d14aa360)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360"
Jul 16 22:44:26.671052 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:26.670981    2555 scope.go:117] "RemoveContainer" containerID="28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7"
Jul 16 22:44:27.412139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:27.412097    2555 generic.go:356] "Generic (PLEG): container finished" podID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerID="8b88f45702a35e3990a068579fc3db72d73522d21841ed796976ef7e26421ec8" exitCode=6
Jul 16 22:44:27.412334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:27.412168    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" event={"ID":"bd3c465e-a15c-4f2c-9afc-49a7d306bc92","Type":"ContainerDied","Data":"8b88f45702a35e3990a068579fc3db72d73522d21841ed796976ef7e26421ec8"}
Jul 16 22:44:27.412334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:27.412239    2555 scope.go:117] "RemoveContainer" containerID="28405c3ab29c5658ce3aa9e3fbd094a8407ef8b9a2c16082593989c3d2e749a7"
Jul 16 22:44:27.412582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:27.412549    2555 scope.go:117] "RemoveContainer" containerID="8b88f45702a35e3990a068579fc3db72d73522d21841ed796976ef7e26421ec8"
Jul 16 22:44:27.412759 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:27.412741    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-s897r_odh-ai-gateway-infra(bd3c465e-a15c-4f2c-9afc-49a7d306bc92)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92"
Jul 16 22:44:28.443945 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:28.443916    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:28.585456 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:28.585433    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:28.619739 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:28.619708    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca\") pod \"bd3c465e-a15c-4f2c-9afc-49a7d306bc92\" (UID: \"bd3c465e-a15c-4f2c-9afc-49a7d306bc92\") "
Jul 16 22:44:28.620135 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:28.620104    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bd3c465e-a15c-4f2c-9afc-49a7d306bc92" (UID: "bd3c465e-a15c-4f2c-9afc-49a7d306bc92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:28.721173 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:28.721080    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bd3c465e-a15c-4f2c-9afc-49a7d306bc92-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:44:29.425772 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.425744    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"
Jul 16 22:44:29.425927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.425791    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r" event={"ID":"bd3c465e-a15c-4f2c-9afc-49a7d306bc92","Type":"ContainerDied","Data":"7e6d35a8130c3e6dfc54bd17a6fb8ffa81434256faf5d98cd548b71c69fded53"}
Jul 16 22:44:29.425927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.425852    2555 scope.go:117] "RemoveContainer" containerID="8b88f45702a35e3990a068579fc3db72d73522d21841ed796976ef7e26421ec8"
Jul 16 22:44:29.450429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.450402    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:29.457017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.456995    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-s897r"]
Jul 16 22:44:29.676528 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.676417    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" path="/var/lib/kubelet/pods/bd3c465e-a15c-4f2c-9afc-49a7d306bc92/volumes"
Jul 16 22:44:29.946878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.946796    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:29.947186 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947173    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947188    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947205    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947214    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947226    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947231 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947231    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947300    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.947376 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.947307    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:29.951818 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.951800    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:29.957966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:29.957941    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:30.032537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.032506    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx\" (UID: \"ad5fec50-6024-4295-954c-5717381f80b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:30.132975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.132936    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx\" (UID: \"ad5fec50-6024-4295-954c-5717381f80b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:30.133566 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.133546    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx\" (UID: \"ad5fec50-6024-4295-954c-5717381f80b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:30.263744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.263644    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:30.408890 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.408861    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:30.433714 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:30.433690    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" event={"ID":"ad5fec50-6024-4295-954c-5717381f80b1","Type":"ContainerStarted","Data":"fd5eaa8eb2ebb1cbc66601e75aa8257f0c85b3b59f9eeefeeef7bd7091f0f0e2"}
Jul 16 22:44:31.073794 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.073765    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:31.074193 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.074180    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd3c465e-a15c-4f2c-9afc-49a7d306bc92" containerName="revoke-keys"
Jul 16 22:44:31.077139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.077124    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:31.086280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.086255    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:31.144450 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.144410    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf\" (UID: \"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:31.245492 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.245431    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf\" (UID: \"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:31.246048 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.246029    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf\" (UID: \"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:31.389305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.389270    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:31.441386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.441351    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad5fec50-6024-4295-954c-5717381f80b1" containerID="a82845dad9067846842d897f78c4dc35e1d08fe2982a950cb13d89391bddef3c" exitCode=6
Jul 16 22:44:31.441577 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.441453    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" event={"ID":"ad5fec50-6024-4295-954c-5717381f80b1","Type":"ContainerDied","Data":"a82845dad9067846842d897f78c4dc35e1d08fe2982a950cb13d89391bddef3c"}
Jul 16 22:44:31.441672 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.441653    2555 scope.go:117] "RemoveContainer" containerID="a82845dad9067846842d897f78c4dc35e1d08fe2982a950cb13d89391bddef3c"
Jul 16 22:44:31.536928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:31.536895    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:31.538562 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:44:31.538530    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5efe78a_ca6b_43a9_8e4e_a5b2face90f6.slice/crio-792e064e509b7de57b87b4b2556c64e6bdfdb66b17cde571ec2473155a705f8f WatchSource:0}: Error finding container 792e064e509b7de57b87b4b2556c64e6bdfdb66b17cde571ec2473155a705f8f: Status 404 returned error can't find the container with id 792e064e509b7de57b87b4b2556c64e6bdfdb66b17cde571ec2473155a705f8f
Jul 16 22:44:32.448635 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.448596    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerID="617ddc310bdce9c1305fff2007aa756d4d09112dfdbe6bf151fa6f99c8df6b67" exitCode=6
Jul 16 22:44:32.449068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.448685    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" event={"ID":"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6","Type":"ContainerDied","Data":"617ddc310bdce9c1305fff2007aa756d4d09112dfdbe6bf151fa6f99c8df6b67"}
Jul 16 22:44:32.449068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.448743    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" event={"ID":"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6","Type":"ContainerStarted","Data":"792e064e509b7de57b87b4b2556c64e6bdfdb66b17cde571ec2473155a705f8f"}
Jul 16 22:44:32.449068 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.448875    2555 scope.go:117] "RemoveContainer" containerID="617ddc310bdce9c1305fff2007aa756d4d09112dfdbe6bf151fa6f99c8df6b67"
Jul 16 22:44:32.450663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.450643    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad5fec50-6024-4295-954c-5717381f80b1" containerID="8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202" exitCode=6
Jul 16 22:44:32.450754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.450692    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" event={"ID":"ad5fec50-6024-4295-954c-5717381f80b1","Type":"ContainerDied","Data":"8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202"}
Jul 16 22:44:32.450754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.450730    2555 scope.go:117] "RemoveContainer" containerID="a82845dad9067846842d897f78c4dc35e1d08fe2982a950cb13d89391bddef3c"
Jul 16 22:44:32.450940 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:32.450920    2555 scope.go:117] "RemoveContainer" containerID="8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202"
Jul 16 22:44:32.451170 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:32.451150    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx_odh-ai-gateway-infra(ad5fec50-6024-4295-954c-5717381f80b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" podUID="ad5fec50-6024-4295-954c-5717381f80b1"
Jul 16 22:44:33.458869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:33.458827    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerID="019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9" exitCode=6
Jul 16 22:44:33.459339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:33.458899    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" event={"ID":"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6","Type":"ContainerDied","Data":"019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9"}
Jul 16 22:44:33.459339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:33.458962    2555 scope.go:117] "RemoveContainer" containerID="617ddc310bdce9c1305fff2007aa756d4d09112dfdbe6bf151fa6f99c8df6b67"
Jul 16 22:44:33.459339 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:33.459156    2555 scope.go:117] "RemoveContainer" containerID="019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9"
Jul 16 22:44:33.459513 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:33.459391    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf_odh-ai-gateway-infra(d5efe78a-ca6b-43a9-8e4e-a5b2face90f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6"
Jul 16 22:44:33.460883 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:33.460869    2555 scope.go:117] "RemoveContainer" containerID="8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202"
Jul 16 22:44:33.461025 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:33.461008    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx_odh-ai-gateway-infra(ad5fec50-6024-4295-954c-5717381f80b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" podUID="ad5fec50-6024-4295-954c-5717381f80b1"
Jul 16 22:44:34.469344 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:34.469317    2555 scope.go:117] "RemoveContainer" containerID="019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9"
Jul 16 22:44:34.469753 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:34.469540    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf_odh-ai-gateway-infra(d5efe78a-ca6b-43a9-8e4e-a5b2face90f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6"
Jul 16 22:44:35.673544 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:35.673506    2555 scope.go:117] "RemoveContainer" containerID="0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61"
Jul 16 22:44:36.487616 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:36.487586    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerID="f6730df8a8b5457299e6ccaef5cf2299781a513b9df6cbd1bfd5a79d6592b9dc" exitCode=6
Jul 16 22:44:36.487791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:36.487666    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" event={"ID":"d5b32778-ae14-4b6a-9fd2-2565d14aa360","Type":"ContainerDied","Data":"f6730df8a8b5457299e6ccaef5cf2299781a513b9df6cbd1bfd5a79d6592b9dc"}
Jul 16 22:44:36.487791 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:36.487731    2555 scope.go:117] "RemoveContainer" containerID="0e7235ea1dea433e70d1a9ae5c31eb091e7ec5fd34dc434de8e9c358301d9f61"
Jul 16 22:44:36.488063 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:36.488047    2555 scope.go:117] "RemoveContainer" containerID="f6730df8a8b5457299e6ccaef5cf2299781a513b9df6cbd1bfd5a79d6592b9dc"
Jul 16 22:44:36.488314 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:36.488289    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r_odh-ai-gateway-infra(d5b32778-ae14-4b6a-9fd2-2565d14aa360)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360"
Jul 16 22:44:37.517195 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:37.517166    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:37.653803 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:37.653779    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:37.708328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:37.708303    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca\") pod \"d5b32778-ae14-4b6a-9fd2-2565d14aa360\" (UID: \"d5b32778-ae14-4b6a-9fd2-2565d14aa360\") "
Jul 16 22:44:37.708690 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:37.708669    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5b32778-ae14-4b6a-9fd2-2565d14aa360" (UID: "d5b32778-ae14-4b6a-9fd2-2565d14aa360"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:37.809749 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:37.809716    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5b32778-ae14-4b6a-9fd2-2565d14aa360-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:44:38.501858 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:38.501832    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"
Jul 16 22:44:38.502031 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:38.501831    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r" event={"ID":"d5b32778-ae14-4b6a-9fd2-2565d14aa360","Type":"ContainerDied","Data":"df1dd733af9d452b1ba542b56369d49e81ed618f6b9f666e20d90f254c07febe"}
Jul 16 22:44:38.502031 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:38.501969    2555 scope.go:117] "RemoveContainer" containerID="f6730df8a8b5457299e6ccaef5cf2299781a513b9df6cbd1bfd5a79d6592b9dc"
Jul 16 22:44:38.525627 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:38.525602    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:38.540771 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:38.540748    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-6pb6r"]
Jul 16 22:44:39.347359 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:39.347324    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:44:39.676214 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:39.676125    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" path="/var/lib/kubelet/pods/d5b32778-ae14-4b6a-9fd2-2565d14aa360/volumes"
Jul 16 22:44:45.673699 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:45.673669    2555 scope.go:117] "RemoveContainer" containerID="019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9"
Jul 16 22:44:45.676028 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:45.673728    2555 scope.go:117] "RemoveContainer" containerID="8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202"
Jul 16 22:44:46.548258 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.548224    2555 generic.go:356] "Generic (PLEG): container finished" podID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerID="03a10709c88544d5030da38c0921ee31c28b9d4a76f5647c93bcf1885c08978c" exitCode=6
Jul 16 22:44:46.548451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.548301    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" event={"ID":"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6","Type":"ContainerDied","Data":"03a10709c88544d5030da38c0921ee31c28b9d4a76f5647c93bcf1885c08978c"}
Jul 16 22:44:46.548451 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.548372    2555 scope.go:117] "RemoveContainer" containerID="019a4b946f54c79f3d553a6e0c92ac7a463672af0c6bfb0804074113a4b718f9"
Jul 16 22:44:46.548747 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.548720    2555 scope.go:117] "RemoveContainer" containerID="03a10709c88544d5030da38c0921ee31c28b9d4a76f5647c93bcf1885c08978c"
Jul 16 22:44:46.549001 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:46.548970    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf_odh-ai-gateway-infra(d5efe78a-ca6b-43a9-8e4e-a5b2face90f6)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6"
Jul 16 22:44:46.550290 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.550272    2555 generic.go:356] "Generic (PLEG): container finished" podID="ad5fec50-6024-4295-954c-5717381f80b1" containerID="4eaecf45ab080c1e527429cc9b329712c3eb4204205a63b98dbba1397ee0f21e" exitCode=6
Jul 16 22:44:46.550383 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.550320    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" event={"ID":"ad5fec50-6024-4295-954c-5717381f80b1","Type":"ContainerDied","Data":"4eaecf45ab080c1e527429cc9b329712c3eb4204205a63b98dbba1397ee0f21e"}
Jul 16 22:44:46.550612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.550600    2555 scope.go:117] "RemoveContainer" containerID="4eaecf45ab080c1e527429cc9b329712c3eb4204205a63b98dbba1397ee0f21e"
Jul 16 22:44:46.550769 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:44:46.550754    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx_odh-ai-gateway-infra(ad5fec50-6024-4295-954c-5717381f80b1)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" podUID="ad5fec50-6024-4295-954c-5717381f80b1"
Jul 16 22:44:46.559797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:46.559777    2555 scope.go:117] "RemoveContainer" containerID="8149a52cc3ee3727dee6aabb2d25e68653be0bac3e06cd59cdc5c0d684833202"
Jul 16 22:44:47.599796 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.599756    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:47.612456 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.612430    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:47.758550 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.758529    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:47.760723 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.760705    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:47.900342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.900256    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca\") pod \"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6\" (UID: \"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6\") "
Jul 16 22:44:47.900342 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.900292    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca\") pod \"ad5fec50-6024-4295-954c-5717381f80b1\" (UID: \"ad5fec50-6024-4295-954c-5717381f80b1\") "
Jul 16 22:44:47.900659 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.900636    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" (UID: "d5efe78a-ca6b-43a9-8e4e-a5b2face90f6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:47.900704 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:47.900660    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ad5fec50-6024-4295-954c-5717381f80b1" (UID: "ad5fec50-6024-4295-954c-5717381f80b1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:44:48.001132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.001092    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:44:48.001132 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.001126    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad5fec50-6024-4295-954c-5717381f80b1-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:44:48.565563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.565531    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"
Jul 16 22:44:48.565751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.565566    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx" event={"ID":"ad5fec50-6024-4295-954c-5717381f80b1","Type":"ContainerDied","Data":"fd5eaa8eb2ebb1cbc66601e75aa8257f0c85b3b59f9eeefeeef7bd7091f0f0e2"}
Jul 16 22:44:48.565751 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.565626    2555 scope.go:117] "RemoveContainer" containerID="4eaecf45ab080c1e527429cc9b329712c3eb4204205a63b98dbba1397ee0f21e"
Jul 16 22:44:48.567048 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.567022    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf" event={"ID":"d5efe78a-ca6b-43a9-8e4e-a5b2face90f6","Type":"ContainerDied","Data":"792e064e509b7de57b87b4b2556c64e6bdfdb66b17cde571ec2473155a705f8f"}
Jul 16 22:44:48.567153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.567057    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"
Jul 16 22:44:48.574825 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.574804    2555 scope.go:117] "RemoveContainer" containerID="03a10709c88544d5030da38c0921ee31c28b9d4a76f5647c93bcf1885c08978c"
Jul 16 22:44:48.608458 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.608431    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:48.622363 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.622340    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-mqkkx"]
Jul 16 22:44:48.666374 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.666338    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:48.684118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:48.684081    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-x7rcf"]
Jul 16 22:44:49.676634 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:49.676597    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad5fec50-6024-4295-954c-5717381f80b1" path="/var/lib/kubelet/pods/ad5fec50-6024-4295-954c-5717381f80b1/volumes"
Jul 16 22:44:49.677001 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:49.676944    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" path="/var/lib/kubelet/pods/d5efe78a-ca6b-43a9-8e4e-a5b2face90f6/volumes"
Jul 16 22:44:55.885833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:44:55.885800    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:45:00.170384 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170348    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"]
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170707    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170720    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170734    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170741    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170748    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170753    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170769    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170774    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170781    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170786    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170793    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170797    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170806    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.170827 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170810    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170863    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170871    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170876    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170883    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170888    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170895    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.171281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.170899    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.175596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.175567    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"
Jul 16 22:45:00.180537 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.180518    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:45:00.184809 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.184793    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.196959    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"]
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197397    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197411    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197421    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197426    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5efe78a-ca6b-43a9-8e4e-a5b2face90f6" containerName="revoke-keys"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197527    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad5fec50-6024-4295-954c-5717381f80b1" containerName="revoke-keys"
Jul 16 22:45:00.200546 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.197538    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5b32778-ae14-4b6a-9fd2-2565d14aa360" containerName="revoke-keys"
Jul 16 22:45:00.200975 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.200780    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"
Jul 16 22:45:00.210972 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.210951    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"]
Jul 16 22:45:00.215440 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.215418    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"]
Jul 16 22:45:00.215579 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.215535    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"
Jul 16 22:45:00.222985 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.222965    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"
Jul 16 22:45:00.230086 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.230071    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"
Jul 16 22:45:00.250089 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.249886    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"]
Jul 16 22:45:00.290005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.289928    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"]
Jul 16 22:45:00.400025 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.399991    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"]
Jul 16 22:45:00.401385 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:00.401354    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79c140c4_3a1e_4d05_9f0b_7a0a56ad03ab.slice/crio-6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2 WatchSource:0}: Error finding container 6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2: Status 404 returned error can't find the container with id 6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2
Jul 16 22:45:00.440622 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.440597    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"]
Jul 16 22:45:00.441718 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:00.441693    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5aa46f7d_5485_4a79_9537_7715257ca5f0.slice/crio-2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69 WatchSource:0}: Error finding container 2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69: Status 404 returned error can't find the container with id 2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69
Jul 16 22:45:00.442626 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:00.442605    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2f02d9_fb22_40a8_8439_77888017b488.slice/crio-bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec WatchSource:0}: Error finding container bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec: Status 404 returned error can't find the container with id bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec
Jul 16 22:45:00.443948 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.443913    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"]
Jul 16 22:45:00.635955 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.635913    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9" event={"ID":"79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab","Type":"ContainerStarted","Data":"6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2"}
Jul 16 22:45:00.637182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.637146    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md" event={"ID":"5aa46f7d-5485-4a79-9537-7715257ca5f0","Type":"ContainerStarted","Data":"2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69"}
Jul 16 22:45:00.638455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:00.638433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd" event={"ID":"3e2f02d9-fb22-40a8-8439-77888017b488","Type":"ContainerStarted","Data":"bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec"}
Jul 16 22:45:01.645752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.645666    2555 generic.go:356] "Generic (PLEG): container finished" podID="79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab" containerID="63077d4a745c655de7c0c1712818372f19d9c613db97fc06434a8ee7006b8012" exitCode=0
Jul 16 22:45:01.646178 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.645746    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9" event={"ID":"79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab","Type":"ContainerDied","Data":"63077d4a745c655de7c0c1712818372f19d9c613db97fc06434a8ee7006b8012"}
Jul 16 22:45:01.647210 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.647184    2555 generic.go:356] "Generic (PLEG): container finished" podID="5aa46f7d-5485-4a79-9537-7715257ca5f0" containerID="67f04188ea967b94b4ac885dc91ec31ad247ff6d664cf36c61e807b785b51909" exitCode=0
Jul 16 22:45:01.647334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.647268    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md" event={"ID":"5aa46f7d-5485-4a79-9537-7715257ca5f0","Type":"ContainerDied","Data":"67f04188ea967b94b4ac885dc91ec31ad247ff6d664cf36c61e807b785b51909"}
Jul 16 22:45:01.648718 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.648699    2555 generic.go:356] "Generic (PLEG): container finished" podID="3e2f02d9-fb22-40a8-8439-77888017b488" containerID="63d48e3de645b7beccbdd5b9e62ca9f07d9905c6d9d3a9474a098a35a50d910c" exitCode=0
Jul 16 22:45:01.648824 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.648748    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd" event={"ID":"3e2f02d9-fb22-40a8-8439-77888017b488","Type":"ContainerDied","Data":"63d48e3de645b7beccbdd5b9e62ca9f07d9905c6d9d3a9474a098a35a50d910c"}
Jul 16 22:45:01.714842 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.714810    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:01.718228 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.718213    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:01.721287 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.721268    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:45:01.727563 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.727531    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:01.729406 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.729380    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6gqsr\" (UID: \"e0c4927e-d89a-4951-ad8c-45b343b389cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:01.830156 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.830122    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6gqsr\" (UID: \"e0c4927e-d89a-4951-ad8c-45b343b389cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:01.830726 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:01.830698    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-6gqsr\" (UID: \"e0c4927e-d89a-4951-ad8c-45b343b389cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:02.029166 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.029141    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:02.169369 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.169338    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:02.170974 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:02.170949    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0c4927e_d89a_4951_ad8c_45b343b389cc.slice/crio-95777dce490cd75fe0a22a34aca2e8dd17b1ca039070460ce001fa658b35de98 WatchSource:0}: Error finding container 95777dce490cd75fe0a22a34aca2e8dd17b1ca039070460ce001fa658b35de98: Status 404 returned error can't find the container with id 95777dce490cd75fe0a22a34aca2e8dd17b1ca039070460ce001fa658b35de98
Jul 16 22:45:02.656297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.656185    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerStarted","Data":"370b553d8ae8169d18490354ca76d0aee9c1526117aea0a20019e2b5d17b2c99"}
Jul 16 22:45:02.656297 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.656255    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerStarted","Data":"95777dce490cd75fe0a22a34aca2e8dd17b1ca039070460ce001fa658b35de98"}
Jul 16 22:45:02.681900 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.681831    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" podStartSLOduration=1.68180921 podStartE2EDuration="1.68180921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:02.677324625 +0000 UTC m=+3357.514378425" watchObservedRunningTime="2026-07-16 22:45:02.68180921 +0000 UTC m=+3357.518863011"
Jul 16 22:45:02.805966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.805941    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"
Jul 16 22:45:02.865403 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.865373    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"
Jul 16 22:45:02.869017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:02.868988    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"
Jul 16 22:45:03.530407 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.530375    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:45:03.663854 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.663821    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"
Jul 16 22:45:03.664280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.663864    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd" event={"ID":"3e2f02d9-fb22-40a8-8439-77888017b488","Type":"ContainerDied","Data":"bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec"}
Jul 16 22:45:03.664280 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.663924    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc09671591b4c2a4f45aa08d10970d506bf354230bfc5f4e15ee4acd6c107eec"
Jul 16 22:45:03.665331 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.665300    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"
Jul 16 22:45:03.665484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.665329    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9" event={"ID":"79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab","Type":"ContainerDied","Data":"6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2"}
Jul 16 22:45:03.665484 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.665372    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d1a6c6437543da913b71e7d5df2654a0be6d03f315db763f5c8446c0046abb2"
Jul 16 22:45:03.666823 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.666787    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md" event={"ID":"5aa46f7d-5485-4a79-9537-7715257ca5f0","Type":"ContainerDied","Data":"2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69"}
Jul 16 22:45:03.666823 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.666807    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29737365-g29md"
Jul 16 22:45:03.666823 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.666826    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e8b530030920a01f1df012af493089af98fdb86e1059929bdde0165d67efb69"
Jul 16 22:45:03.668505 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.668455    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerID="370b553d8ae8169d18490354ca76d0aee9c1526117aea0a20019e2b5d17b2c99" exitCode=6
Jul 16 22:45:03.668604 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.668511    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerDied","Data":"370b553d8ae8169d18490354ca76d0aee9c1526117aea0a20019e2b5d17b2c99"}
Jul 16 22:45:03.668719 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:03.668706    2555 scope.go:117] "RemoveContainer" containerID="370b553d8ae8169d18490354ca76d0aee9c1526117aea0a20019e2b5d17b2c99"
Jul 16 22:45:04.675905 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:04.675818    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerID="0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45" exitCode=6
Jul 16 22:45:04.676322 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:04.675900    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerDied","Data":"0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45"}
Jul 16 22:45:04.676322 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:04.675958    2555 scope.go:117] "RemoveContainer" containerID="370b553d8ae8169d18490354ca76d0aee9c1526117aea0a20019e2b5d17b2c99"
Jul 16 22:45:04.676322 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:04.676099    2555 scope.go:117] "RemoveContainer" containerID="0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45"
Jul 16 22:45:04.676322 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:04.676298    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6gqsr_odh-ai-gateway-infra(e0c4927e-d89a-4951-ad8c-45b343b389cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc"
Jul 16 22:45:05.683836 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:05.683805    2555 scope.go:117] "RemoveContainer" containerID="0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45"
Jul 16 22:45:05.684203 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:05.684019    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6gqsr_odh-ai-gateway-infra(e0c4927e-d89a-4951-ad8c-45b343b389cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc"
Jul 16 22:45:10.044485 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044428    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044825    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5aa46f7d-5485-4a79-9537-7715257ca5f0" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044841    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5aa46f7d-5485-4a79-9537-7715257ca5f0" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044855    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044863    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044879    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2f02d9-fb22-40a8-8439-77888017b488" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044887    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2f02d9-fb22-40a8-8439-77888017b488" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044960    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044972    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5aa46f7d-5485-4a79-9537-7715257ca5f0" containerName="cleanup"
Jul 16 22:45:10.045005 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.044984    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2f02d9-fb22-40a8-8439-77888017b488" containerName="cleanup"
Jul 16 22:45:10.049455 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.049439    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:10.057951 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.057922    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:10.103120 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.103081    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z\" (UID: \"ae13c3e9-2527-48bc-8a47-7614897b3f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:10.204433 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.204398    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z\" (UID: \"ae13c3e9-2527-48bc-8a47-7614897b3f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:10.205001 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.204983    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z\" (UID: \"ae13c3e9-2527-48bc-8a47-7614897b3f4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:10.361502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.361385    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:10.509179 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.509147    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:10.509298 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:10.509256    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae13c3e9_2527_48bc_8a47_7614897b3f4c.slice/crio-2386e93252d4daf4ec77ddda48f1cfba573ec337fe3dc5e33637672bb1529447 WatchSource:0}: Error finding container 2386e93252d4daf4ec77ddda48f1cfba573ec337fe3dc5e33637672bb1529447: Status 404 returned error can't find the container with id 2386e93252d4daf4ec77ddda48f1cfba573ec337fe3dc5e33637672bb1529447
Jul 16 22:45:10.712559 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:10.712485    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" event={"ID":"ae13c3e9-2527-48bc-8a47-7614897b3f4c","Type":"ContainerStarted","Data":"2386e93252d4daf4ec77ddda48f1cfba573ec337fe3dc5e33637672bb1529447"}
Jul 16 22:45:11.102752 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:11.102718    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:45:11.719666 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:11.719630    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerID="447f7b339b9a4a498dd568c7f30b153ce2fbcfd63f1e8ee1df24708a93a222d5" exitCode=6
Jul 16 22:45:11.719861 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:11.719725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" event={"ID":"ae13c3e9-2527-48bc-8a47-7614897b3f4c","Type":"ContainerDied","Data":"447f7b339b9a4a498dd568c7f30b153ce2fbcfd63f1e8ee1df24708a93a222d5"}
Jul 16 22:45:11.719921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:11.719895    2555 scope.go:117] "RemoveContainer" containerID="447f7b339b9a4a498dd568c7f30b153ce2fbcfd63f1e8ee1df24708a93a222d5"
Jul 16 22:45:12.728452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:12.728359    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerID="f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d" exitCode=6
Jul 16 22:45:12.728452 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:12.728437    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" event={"ID":"ae13c3e9-2527-48bc-8a47-7614897b3f4c","Type":"ContainerDied","Data":"f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d"}
Jul 16 22:45:12.728935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:12.728513    2555 scope.go:117] "RemoveContainer" containerID="447f7b339b9a4a498dd568c7f30b153ce2fbcfd63f1e8ee1df24708a93a222d5"
Jul 16 22:45:12.728935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:12.728747    2555 scope.go:117] "RemoveContainer" containerID="f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d"
Jul 16 22:45:12.729015 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:12.728983    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z_odh-ai-gateway-infra(ae13c3e9-2527-48bc-8a47-7614897b3f4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c"
Jul 16 22:45:13.735846 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:13.735820    2555 scope.go:117] "RemoveContainer" containerID="f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d"
Jul 16 22:45:13.736240 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:13.736028    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z_odh-ai-gateway-infra(ae13c3e9-2527-48bc-8a47-7614897b3f4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c"
Jul 16 22:45:18.551637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.551588    2555 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" podUID="aa829904-3666-434c-bb9b-b89e69bfaf42" containerName="main" containerID="cri-o://c2bed011ffe38f988b15ac3d5987ec089d8799388dc5967112e8b7bf91794091" gracePeriod=45
Jul 16 22:45:18.766316 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.766283    2555 generic.go:356] "Generic (PLEG): container finished" podID="aa829904-3666-434c-bb9b-b89e69bfaf42" containerID="c2bed011ffe38f988b15ac3d5987ec089d8799388dc5967112e8b7bf91794091" exitCode=0
Jul 16 22:45:18.766449 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.766354    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" event={"ID":"aa829904-3666-434c-bb9b-b89e69bfaf42","Type":"ContainerDied","Data":"c2bed011ffe38f988b15ac3d5987ec089d8799388dc5967112e8b7bf91794091"}
Jul 16 22:45:18.806821 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.806763    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:45:18.885932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.885895    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886128 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.885977    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886128 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886084    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886150    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7pfx9\" (UniqueName: \"kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886190    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886223    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache" (OuterVolumeSpecName: "model-cache") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:18.886250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886229    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir\") pod \"aa829904-3666-434c-bb9b-b89e69bfaf42\" (UID: \"aa829904-3666-434c-bb9b-b89e69bfaf42\") "
Jul 16 22:45:18.886435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886337    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home" (OuterVolumeSpecName: "home") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:18.886435 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886410    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:18.886573 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886561    2555 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-tmp-dir\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:18.886639 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886575    2555 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-model-cache\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:18.886639 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.886587    2555 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-home\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:18.888426 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.888404    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9" (OuterVolumeSpecName: "kube-api-access-7pfx9") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "kube-api-access-7pfx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:45:18.888522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.888406    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm" (OuterVolumeSpecName: "dshm") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 22:45:18.888522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.888447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "aa829904-3666-434c-bb9b-b89e69bfaf42" (UID: "aa829904-3666-434c-bb9b-b89e69bfaf42"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:45:18.987865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.987817    2555 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7pfx9\" (UniqueName: \"kubernetes.io/projected/aa829904-3666-434c-bb9b-b89e69bfaf42-kube-api-access-7pfx9\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:18.987865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.987859    2555 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/aa829904-3666-434c-bb9b-b89e69bfaf42-dshm\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:18.987865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:18.987871    2555 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/aa829904-3666-434c-bb9b-b89e69bfaf42-tls-certs\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:19.670857 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.670828    2555 scope.go:117] "RemoveContainer" containerID="0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45"
Jul 16 22:45:19.773433 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.773402    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"
Jul 16 22:45:19.773598 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.773433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69" event={"ID":"aa829904-3666-434c-bb9b-b89e69bfaf42","Type":"ContainerDied","Data":"f77c394b0bbc0400279f495d3c54bd1f18b936501d14e7478a090c5abfc3ee19"}
Jul 16 22:45:19.773598 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.773500    2555 scope.go:117] "RemoveContainer" containerID="c2bed011ffe38f988b15ac3d5987ec089d8799388dc5967112e8b7bf91794091"
Jul 16 22:45:19.793553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.793530    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:45:19.798213 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:19.798192    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-bb55e2/rate-test-model-bb55e2-kserve-5fcff7b57f-f4v69"]
Jul 16 22:45:20.004839 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.004802    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:20.005308 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.005295    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aa829904-3666-434c-bb9b-b89e69bfaf42" containerName="main"
Jul 16 22:45:20.005356 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.005310    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="aa829904-3666-434c-bb9b-b89e69bfaf42" containerName="main"
Jul 16 22:45:20.005394 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.005372    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="aa829904-3666-434c-bb9b-b89e69bfaf42" containerName="main"
Jul 16 22:45:20.008262 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.008246    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:20.017367 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.017343    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:20.098190 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.098151    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r\" (UID: \"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:20.199317 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.199230    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r\" (UID: \"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:20.199867 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.199848    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r\" (UID: \"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:20.320184 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.320151    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:20.461663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.461631    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:20.462712 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:20.462687    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dfc4c17_8920_4e22_ac5b_6d877edcd1f5.slice/crio-622baa7289b305e6572685073e6e37f42fb84a7f8f57d616f263482308dc8548 WatchSource:0}: Error finding container 622baa7289b305e6572685073e6e37f42fb84a7f8f57d616f263482308dc8548: Status 404 returned error can't find the container with id 622baa7289b305e6572685073e6e37f42fb84a7f8f57d616f263482308dc8548
Jul 16 22:45:20.781149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.781117    2555 generic.go:356] "Generic (PLEG): container finished" podID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerID="6126a2202ab37fe8c8990206ba5edf3a51745667f01de1bda2d059f4468cb8bf" exitCode=6
Jul 16 22:45:20.781625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.781191    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerDied","Data":"6126a2202ab37fe8c8990206ba5edf3a51745667f01de1bda2d059f4468cb8bf"}
Jul 16 22:45:20.781625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.781257    2555 scope.go:117] "RemoveContainer" containerID="0615881afe417fd91594c617de969394296c2dfc567d1f1de2a236277ceb8d45"
Jul 16 22:45:20.781625 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.781573    2555 scope.go:117] "RemoveContainer" containerID="6126a2202ab37fe8c8990206ba5edf3a51745667f01de1bda2d059f4468cb8bf"
Jul 16 22:45:20.781836 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:20.781814    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-6gqsr_odh-ai-gateway-infra(e0c4927e-d89a-4951-ad8c-45b343b389cc)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc"
Jul 16 22:45:20.784091 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.784058    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerStarted","Data":"e2f441afaa016bfe6e2c36ed46f44e1526eeb9b0316b2bb2cdfc951331bca3af"}
Jul 16 22:45:20.784251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.784121    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerStarted","Data":"622baa7289b305e6572685073e6e37f42fb84a7f8f57d616f263482308dc8548"}
Jul 16 22:45:20.858541 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:20.858478    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" podStartSLOduration=1.858450639 podStartE2EDuration="1.858450639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:45:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:45:20.856351578 +0000 UTC m=+3375.693405377" watchObservedRunningTime="2026-07-16 22:45:20.858450639 +0000 UTC m=+3375.695504418"
Jul 16 22:45:21.143311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.143278    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:21.146627 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.146612    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:21.159272 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.159241    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:21.207098 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.207058    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v\" (UID: \"a0900f45-2c6f-4c39-a83a-073ef8baaaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:21.307928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.307890    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v\" (UID: \"a0900f45-2c6f-4c39-a83a-073ef8baaaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:21.308553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.308535    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v\" (UID: \"a0900f45-2c6f-4c39-a83a-073ef8baaaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:21.457599 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.457564    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:21.612866 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.612787    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:21.616996 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:21.616969    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0900f45_2c6f_4c39_a83a_073ef8baaaa3.slice/crio-9a5e88a11eb152a1d9d9b44a59285b770460ebeeb27daa997d23a78a280d5d78 WatchSource:0}: Error finding container 9a5e88a11eb152a1d9d9b44a59285b770460ebeeb27daa997d23a78a280d5d78: Status 404 returned error can't find the container with id 9a5e88a11eb152a1d9d9b44a59285b770460ebeeb27daa997d23a78a280d5d78
Jul 16 22:45:21.676029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.676007    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aa829904-3666-434c-bb9b-b89e69bfaf42" path="/var/lib/kubelet/pods/aa829904-3666-434c-bb9b-b89e69bfaf42/volumes"
Jul 16 22:45:21.792808 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.792776    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerID="e2f441afaa016bfe6e2c36ed46f44e1526eeb9b0316b2bb2cdfc951331bca3af" exitCode=6
Jul 16 22:45:21.793243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.792841    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerDied","Data":"e2f441afaa016bfe6e2c36ed46f44e1526eeb9b0316b2bb2cdfc951331bca3af"}
Jul 16 22:45:21.793243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.793019    2555 scope.go:117] "RemoveContainer" containerID="e2f441afaa016bfe6e2c36ed46f44e1526eeb9b0316b2bb2cdfc951331bca3af"
Jul 16 22:45:21.794156 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.794130    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" event={"ID":"a0900f45-2c6f-4c39-a83a-073ef8baaaa3","Type":"ContainerStarted","Data":"9a5e88a11eb152a1d9d9b44a59285b770460ebeeb27daa997d23a78a280d5d78"}
Jul 16 22:45:21.825397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:21.825374    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:22.034291 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.034271    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:22.115150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.115122    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca\") pod \"e0c4927e-d89a-4951-ad8c-45b343b389cc\" (UID: \"e0c4927e-d89a-4951-ad8c-45b343b389cc\") "
Jul 16 22:45:22.115429 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.115408    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e0c4927e-d89a-4951-ad8c-45b343b389cc" (UID: "e0c4927e-d89a-4951-ad8c-45b343b389cc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:22.216815 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.216728    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e0c4927e-d89a-4951-ad8c-45b343b389cc-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:22.801980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.801952    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"
Jul 16 22:45:22.801980 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.801960    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr" event={"ID":"e0c4927e-d89a-4951-ad8c-45b343b389cc","Type":"ContainerDied","Data":"95777dce490cd75fe0a22a34aca2e8dd17b1ca039070460ce001fa658b35de98"}
Jul 16 22:45:22.802522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.802027    2555 scope.go:117] "RemoveContainer" containerID="6126a2202ab37fe8c8990206ba5edf3a51745667f01de1bda2d059f4468cb8bf"
Jul 16 22:45:22.804083 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.804057    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerID="8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960" exitCode=6
Jul 16 22:45:22.804221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.804140    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerDied","Data":"8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960"}
Jul 16 22:45:22.804343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.804328    2555 scope.go:117] "RemoveContainer" containerID="8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960"
Jul 16 22:45:22.804597 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:22.804576    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r_odh-ai-gateway-infra(5dfc4c17-8920-4e22-ac5b-6d877edcd1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5"
Jul 16 22:45:22.805862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.805839    2555 generic.go:356] "Generic (PLEG): container finished" podID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerID="07df8b70ab75092e50cedf3eaa3441be0cdb2d083dbf9518bb964422d6b908b9" exitCode=6
Jul 16 22:45:22.805944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.805902    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" event={"ID":"a0900f45-2c6f-4c39-a83a-073ef8baaaa3","Type":"ContainerDied","Data":"07df8b70ab75092e50cedf3eaa3441be0cdb2d083dbf9518bb964422d6b908b9"}
Jul 16 22:45:22.806048 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.806036    2555 scope.go:117] "RemoveContainer" containerID="07df8b70ab75092e50cedf3eaa3441be0cdb2d083dbf9518bb964422d6b908b9"
Jul 16 22:45:22.816006 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.815985    2555 scope.go:117] "RemoveContainer" containerID="e2f441afaa016bfe6e2c36ed46f44e1526eeb9b0316b2bb2cdfc951331bca3af"
Jul 16 22:45:22.862509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.862485    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:22.866788 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:22.866769    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-6gqsr"]
Jul 16 22:45:23.676232 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.676153    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" path="/var/lib/kubelet/pods/e0c4927e-d89a-4951-ad8c-45b343b389cc/volumes"
Jul 16 22:45:23.814499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.814451    2555 scope.go:117] "RemoveContainer" containerID="8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960"
Jul 16 22:45:23.814929 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:23.814742    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r_odh-ai-gateway-infra(5dfc4c17-8920-4e22-ac5b-6d877edcd1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5"
Jul 16 22:45:23.815878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.815855    2555 generic.go:356] "Generic (PLEG): container finished" podID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerID="55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364" exitCode=6
Jul 16 22:45:23.815965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.815933    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" event={"ID":"a0900f45-2c6f-4c39-a83a-073ef8baaaa3","Type":"ContainerDied","Data":"55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364"}
Jul 16 22:45:23.816017 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.815992    2555 scope.go:117] "RemoveContainer" containerID="07df8b70ab75092e50cedf3eaa3441be0cdb2d083dbf9518bb964422d6b908b9"
Jul 16 22:45:23.816152 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:23.816133    2555 scope.go:117] "RemoveContainer" containerID="55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364"
Jul 16 22:45:23.816354 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:23.816335    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v_odh-ai-gateway-infra(a0900f45-2c6f-4c39-a83a-073ef8baaaa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3"
Jul 16 22:45:24.823172 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:24.823137    2555 scope.go:117] "RemoveContainer" containerID="55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364"
Jul 16 22:45:24.823680 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:24.823402    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v_odh-ai-gateway-infra(a0900f45-2c6f-4c39-a83a-073ef8baaaa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3"
Jul 16 22:45:27.670767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:27.670733    2555 scope.go:117] "RemoveContainer" containerID="f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d"
Jul 16 22:45:28.847475 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:28.847425    2555 generic.go:356] "Generic (PLEG): container finished" podID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerID="ab656a0bd06f70b787e88df63e1f4e10f0bdccbadf4795325238b32f867ddd93" exitCode=6
Jul 16 22:45:28.847917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:28.847498    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" event={"ID":"ae13c3e9-2527-48bc-8a47-7614897b3f4c","Type":"ContainerDied","Data":"ab656a0bd06f70b787e88df63e1f4e10f0bdccbadf4795325238b32f867ddd93"}
Jul 16 22:45:28.847917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:28.847559    2555 scope.go:117] "RemoveContainer" containerID="f95fb6155c79e7148e1b436d62ff117390aa856f3b260e2681fa004fd668ac2d"
Jul 16 22:45:28.847917 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:28.847858    2555 scope.go:117] "RemoveContainer" containerID="ab656a0bd06f70b787e88df63e1f4e10f0bdccbadf4795325238b32f867ddd93"
Jul 16 22:45:28.848096 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:28.848077    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z_odh-ai-gateway-infra(ae13c3e9-2527-48bc-8a47-7614897b3f4c)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c"
Jul 16 22:45:29.880921 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:29.880888    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:30.024044 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.024017    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:30.086530 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.086502    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca\") pod \"ae13c3e9-2527-48bc-8a47-7614897b3f4c\" (UID: \"ae13c3e9-2527-48bc-8a47-7614897b3f4c\") "
Jul 16 22:45:30.086862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.086840    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ae13c3e9-2527-48bc-8a47-7614897b3f4c" (UID: "ae13c3e9-2527-48bc-8a47-7614897b3f4c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:30.187966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.187874    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ae13c3e9-2527-48bc-8a47-7614897b3f4c-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:30.862139 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.862112    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"
Jul 16 22:45:30.862307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.862149    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z" event={"ID":"ae13c3e9-2527-48bc-8a47-7614897b3f4c","Type":"ContainerDied","Data":"2386e93252d4daf4ec77ddda48f1cfba573ec337fe3dc5e33637672bb1529447"}
Jul 16 22:45:30.862307 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.862212    2555 scope.go:117] "RemoveContainer" containerID="ab656a0bd06f70b787e88df63e1f4e10f0bdccbadf4795325238b32f867ddd93"
Jul 16 22:45:30.888458 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.888431    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:30.911103 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:30.911077    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-fhb8z"]
Jul 16 22:45:31.676568 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:31.676533    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" path="/var/lib/kubelet/pods/ae13c3e9-2527-48bc-8a47-7614897b3f4c/volumes"
Jul 16 22:45:34.590879 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:34.590832    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"]
Jul 16 22:45:34.595358 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:34.595336    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-bb55e2-29737365-6pgcd"]
Jul 16 22:45:35.676660 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:35.676624    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2f02d9-fb22-40a8-8439-77888017b488" path="/var/lib/kubelet/pods/3e2f02d9-fb22-40a8-8439-77888017b488/volumes"
Jul 16 22:45:36.670735 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:36.670708    2555 scope.go:117] "RemoveContainer" containerID="55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364"
Jul 16 22:45:37.197295 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:37.197204    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:45:37.903744 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:37.903711    2555 generic.go:356] "Generic (PLEG): container finished" podID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerID="1095a9258613a0cdaf86cd38e6493ae5c8e2235e8c8f400e9b26dd80a2801b48" exitCode=6
Jul 16 22:45:37.903930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:37.903785    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" event={"ID":"a0900f45-2c6f-4c39-a83a-073ef8baaaa3","Type":"ContainerDied","Data":"1095a9258613a0cdaf86cd38e6493ae5c8e2235e8c8f400e9b26dd80a2801b48"}
Jul 16 22:45:37.903930 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:37.903871    2555 scope.go:117] "RemoveContainer" containerID="55d015c22801296da17c37a988eed3a34ba3923c8e1f0e8276933b539aa2e364"
Jul 16 22:45:37.904153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:37.904136    2555 scope.go:117] "RemoveContainer" containerID="1095a9258613a0cdaf86cd38e6493ae5c8e2235e8c8f400e9b26dd80a2801b48"
Jul 16 22:45:37.904391 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:37.904364    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v_odh-ai-gateway-infra(a0900f45-2c6f-4c39-a83a-073ef8baaaa3)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3"
Jul 16 22:45:38.670553 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:38.670525    2555 scope.go:117] "RemoveContainer" containerID="8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960"
Jul 16 22:45:38.934314 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:38.934246    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:39.068373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.068350    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:39.169516 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.169481    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca\") pod \"a0900f45-2c6f-4c39-a83a-073ef8baaaa3\" (UID: \"a0900f45-2c6f-4c39-a83a-073ef8baaaa3\") "
Jul 16 22:45:39.169789 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.169766    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a0900f45-2c6f-4c39-a83a-073ef8baaaa3" (UID: "a0900f45-2c6f-4c39-a83a-073ef8baaaa3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:39.270355 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.270279    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a0900f45-2c6f-4c39-a83a-073ef8baaaa3-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:39.918202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.918165    2555 generic.go:356] "Generic (PLEG): container finished" podID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerID="14c4bcfbdc1f5e90dfa9c708a64522c2fb7d18001762a00aa6b96b994cc98fe2" exitCode=6
Jul 16 22:45:39.918637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.918242    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerDied","Data":"14c4bcfbdc1f5e90dfa9c708a64522c2fb7d18001762a00aa6b96b994cc98fe2"}
Jul 16 22:45:39.918637 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.918325    2555 scope.go:117] "RemoveContainer" containerID="8c32304ce73c5ec9ac7ed2021f6b6041cbf14d8779c403d82d228e75c89d2960"
Jul 16 22:45:39.918739 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.918655    2555 scope.go:117] "RemoveContainer" containerID="14c4bcfbdc1f5e90dfa9c708a64522c2fb7d18001762a00aa6b96b994cc98fe2"
Jul 16 22:45:39.918934 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:45:39.918914    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r_odh-ai-gateway-infra(5dfc4c17-8920-4e22-ac5b-6d877edcd1f5)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5"
Jul 16 22:45:39.919959 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.919939    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"
Jul 16 22:45:39.920046 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.919952    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v" event={"ID":"a0900f45-2c6f-4c39-a83a-073ef8baaaa3","Type":"ContainerDied","Data":"9a5e88a11eb152a1d9d9b44a59285b770460ebeeb27daa997d23a78a280d5d78"}
Jul 16 22:45:39.929932 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.929903    2555 scope.go:117] "RemoveContainer" containerID="1095a9258613a0cdaf86cd38e6493ae5c8e2235e8c8f400e9b26dd80a2801b48"
Jul 16 22:45:39.955340 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.955298    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:39.961445 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:39.961415    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-qx59v"]
Jul 16 22:45:40.956225 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:40.956194    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:41.089496 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.089453    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:41.183897 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.183867    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca\") pod \"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5\" (UID: \"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5\") "
Jul 16 22:45:41.184224 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.184202    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" (UID: "5dfc4c17-8920-4e22-ac5b-6d877edcd1f5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:41.285502 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.285457    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:41.676966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.676879    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" path="/var/lib/kubelet/pods/a0900f45-2c6f-4c39-a83a-073ef8baaaa3/volumes"
Jul 16 22:45:41.935522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.935402    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r" event={"ID":"5dfc4c17-8920-4e22-ac5b-6d877edcd1f5","Type":"ContainerDied","Data":"622baa7289b305e6572685073e6e37f42fb84a7f8f57d616f263482308dc8548"}
Jul 16 22:45:41.935522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.935423    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"
Jul 16 22:45:41.935522 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.935486    2555 scope.go:117] "RemoveContainer" containerID="14c4bcfbdc1f5e90dfa9c708a64522c2fb7d18001762a00aa6b96b994cc98fe2"
Jul 16 22:45:41.955420 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.955383    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:41.962649 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:41.962621    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-h4m2r"]
Jul 16 22:45:43.675860 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:43.675825    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" path="/var/lib/kubelet/pods/5dfc4c17-8920-4e22-ac5b-6d877edcd1f5/volumes"
Jul 16 22:45:50.248397 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248357    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"]
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248757    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248771    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248779    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248788    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248794    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248800    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248805    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248810    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248818    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248823    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248830    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248835    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248841    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248846    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248852    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.248852 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248857    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248864    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248869    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248878    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248884    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248943    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248951    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248957    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248963    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248967    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248972    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248977    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248982    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:45:50.249444 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.248991    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:45:50.251727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.251710    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:50.254200 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.254178    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:45:50.254305 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.254243    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:45:50.261517 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.261491    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"]
Jul 16 22:45:50.367674 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.367637    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm\" (UID: \"6e193ba2-37ff-4d59-acc6-78b13e2f64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:50.468121 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.468075    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm\" (UID: \"6e193ba2-37ff-4d59-acc6-78b13e2f64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:50.468777 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.468753    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm\" (UID: \"6e193ba2-37ff-4d59-acc6-78b13e2f64a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:50.563532 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.563492    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:50.706753 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.706721    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"]
Jul 16 22:45:50.707537 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:45:50.707502    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e193ba2_37ff_4d59_acc6_78b13e2f64a7.slice/crio-306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8 WatchSource:0}: Error finding container 306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8: Status 404 returned error can't find the container with id 306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8
Jul 16 22:45:50.990347 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:50.990308    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm" event={"ID":"6e193ba2-37ff-4d59-acc6-78b13e2f64a7","Type":"ContainerStarted","Data":"306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8"}
Jul 16 22:45:51.997880 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:51.997847    2555 generic.go:356] "Generic (PLEG): container finished" podID="6e193ba2-37ff-4d59-acc6-78b13e2f64a7" containerID="d59ebd2c9077947ab9cf2f3acc67c4c9524f7cbe98a551ff6cc1d554f5c18a02" exitCode=0
Jul 16 22:45:51.998254 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:51.997918    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm" event={"ID":"6e193ba2-37ff-4d59-acc6-78b13e2f64a7","Type":"ContainerDied","Data":"d59ebd2c9077947ab9cf2f3acc67c4c9524f7cbe98a551ff6cc1d554f5c18a02"}
Jul 16 22:45:53.137743 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:53.137717    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:53.189147 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:53.189117    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca\") pod \"6e193ba2-37ff-4d59-acc6-78b13e2f64a7\" (UID: \"6e193ba2-37ff-4d59-acc6-78b13e2f64a7\") "
Jul 16 22:45:53.189491 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:53.189447    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6e193ba2-37ff-4d59-acc6-78b13e2f64a7" (UID: "6e193ba2-37ff-4d59-acc6-78b13e2f64a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:45:53.289878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:53.289847    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6e193ba2-37ff-4d59-acc6-78b13e2f64a7-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:45:54.010883 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:54.010805    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm"
Jul 16 22:45:54.010883 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:54.010818    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm" event={"ID":"6e193ba2-37ff-4d59-acc6-78b13e2f64a7","Type":"ContainerDied","Data":"306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8"}
Jul 16 22:45:54.010883 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:45:54.010861    2555 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="306748ef426c9fe30d8d9200334fd9e74ac759b2cc0a85d9f5ea1655651648c8"
Jul 16 22:46:00.898373 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:00.898294    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"]
Jul 16 22:46:00.901596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:00.901571    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-dbc614-29737365-8hpn9"]
Jul 16 22:46:01.676479 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.676441    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab" path="/var/lib/kubelet/pods/79c140c4-3a1e-4d05-9f0b-7a0a56ad03ab/volumes"
Jul 16 22:46:01.756364 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756326    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:01.756770 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756752    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:46:01.756862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756772    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:46:01.756862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756784    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:46:01.756862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756792    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dfc4c17-8920-4e22-ac5b-6d877edcd1f5" containerName="revoke-keys"
Jul 16 22:46:01.756862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756818    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e193ba2-37ff-4d59-acc6-78b13e2f64a7" containerName="revoke-keys"
Jul 16 22:46:01.756862 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756827    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e193ba2-37ff-4d59-acc6-78b13e2f64a7" containerName="revoke-keys"
Jul 16 22:46:01.757118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756909    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="a0900f45-2c6f-4c39-a83a-073ef8baaaa3" containerName="revoke-keys"
Jul 16 22:46:01.757118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756927    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e193ba2-37ff-4d59-acc6-78b13e2f64a7" containerName="revoke-keys"
Jul 16 22:46:01.757118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756937    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae13c3e9-2527-48bc-8a47-7614897b3f4c" containerName="revoke-keys"
Jul 16 22:46:01.757118 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.756947    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="e0c4927e-d89a-4951-ad8c-45b343b389cc" containerName="revoke-keys"
Jul 16 22:46:01.760426 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.760405    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:01.763016 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.762997    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:46:01.763131 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.763074    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:46:01.769556 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.769532    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:01.865136 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.865095    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-gg7wv\" (UID: \"3cc4acfc-dd60-4c97-8a8f-211aca027317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:01.966092 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.966007    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-gg7wv\" (UID: \"3cc4acfc-dd60-4c97-8a8f-211aca027317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:01.966663 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:01.966642    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-gg7wv\" (UID: \"3cc4acfc-dd60-4c97-8a8f-211aca027317\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:02.073294 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:02.073262    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:02.212294 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:02.212261    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:02.213250 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:46:02.213225    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cc4acfc_dd60_4c97_8a8f_211aca027317.slice/crio-a9477115e28ce1df36b2e3c6b05485280125a69cbd923b042487de932f2df5a9 WatchSource:0}: Error finding container a9477115e28ce1df36b2e3c6b05485280125a69cbd923b042487de932f2df5a9: Status 404 returned error can't find the container with id a9477115e28ce1df36b2e3c6b05485280125a69cbd923b042487de932f2df5a9
Jul 16 22:46:03.063798 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:03.063707    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerID="2c892621ffbaaa06eb71a0caa001e4c29ac6d0855ca64d5396826e80055a3633" exitCode=6
Jul 16 22:46:03.063798 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:03.063767    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerDied","Data":"2c892621ffbaaa06eb71a0caa001e4c29ac6d0855ca64d5396826e80055a3633"}
Jul 16 22:46:03.064204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:03.063804    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerStarted","Data":"a9477115e28ce1df36b2e3c6b05485280125a69cbd923b042487de932f2df5a9"}
Jul 16 22:46:03.064204 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:03.063970    2555 scope.go:117] "RemoveContainer" containerID="2c892621ffbaaa06eb71a0caa001e4c29ac6d0855ca64d5396826e80055a3633"
Jul 16 22:46:04.073675 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:04.073646    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerID="d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc" exitCode=6
Jul 16 22:46:04.074149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:04.073697    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerDied","Data":"d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc"}
Jul 16 22:46:04.074149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:04.073746    2555 scope.go:117] "RemoveContainer" containerID="2c892621ffbaaa06eb71a0caa001e4c29ac6d0855ca64d5396826e80055a3633"
Jul 16 22:46:04.074149 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:04.073970    2555 scope.go:117] "RemoveContainer" containerID="d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc"
Jul 16 22:46:04.074313 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:04.074249    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-gg7wv_odh-ai-gateway-infra(3cc4acfc-dd60-4c97-8a8f-211aca027317)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317"
Jul 16 22:46:04.100245 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:04.100218    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-xl5tk"]
Jul 16 22:46:05.080952 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:05.080927    2555 scope.go:117] "RemoveContainer" containerID="d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc"
Jul 16 22:46:05.081431 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:05.081171    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-gg7wv_odh-ai-gateway-infra(3cc4acfc-dd60-4c97-8a8f-211aca027317)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317"
Jul 16 22:46:11.189737 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.189703    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:11.195281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.195260    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:11.202386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.202357    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:11.245698 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.245672    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm\" (UID: \"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:11.346915 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.346886    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm\" (UID: \"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:11.347514 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.347498    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm\" (UID: \"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:11.507162 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.507072    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:11.657278 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:11.657245    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:11.658870 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:46:11.658841    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6059a2a0_19c2_4f5f_b2a8_83e2acbf8ec9.slice/crio-174bd6e42914e00b734ee6a55aa7fa66c2ea283bf08c09a49e257141cceda518 WatchSource:0}: Error finding container 174bd6e42914e00b734ee6a55aa7fa66c2ea283bf08c09a49e257141cceda518: Status 404 returned error can't find the container with id 174bd6e42914e00b734ee6a55aa7fa66c2ea283bf08c09a49e257141cceda518
Jul 16 22:46:12.122081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:12.121987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerStarted","Data":"b5d023cdf543e38460e17d96481b2a2b6ef95efa8243f72bfaee2b51743933f6"}
Jul 16 22:46:12.122081 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:12.122055    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerStarted","Data":"174bd6e42914e00b734ee6a55aa7fa66c2ea283bf08c09a49e257141cceda518"}
Jul 16 22:46:12.152168 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:12.152116    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" podStartSLOduration=1.152089582 podStartE2EDuration="1.152089582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:12.151936301 +0000 UTC m=+3426.988990101" watchObservedRunningTime="2026-07-16 22:46:12.152089582 +0000 UTC m=+3426.989143381"
Jul 16 22:46:13.128981 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:13.128944    2555 generic.go:356] "Generic (PLEG): container finished" podID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerID="b5d023cdf543e38460e17d96481b2a2b6ef95efa8243f72bfaee2b51743933f6" exitCode=6
Jul 16 22:46:13.129384 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:13.128987    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerDied","Data":"b5d023cdf543e38460e17d96481b2a2b6ef95efa8243f72bfaee2b51743933f6"}
Jul 16 22:46:13.129384 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:13.129260    2555 scope.go:117] "RemoveContainer" containerID="b5d023cdf543e38460e17d96481b2a2b6ef95efa8243f72bfaee2b51743933f6"
Jul 16 22:46:14.136306 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:14.136274    2555 generic.go:356] "Generic (PLEG): container finished" podID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerID="87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300" exitCode=6
Jul 16 22:46:14.136782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:14.136359    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerDied","Data":"87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300"}
Jul 16 22:46:14.136782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:14.136430    2555 scope.go:117] "RemoveContainer" containerID="b5d023cdf543e38460e17d96481b2a2b6ef95efa8243f72bfaee2b51743933f6"
Jul 16 22:46:14.136782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:14.136574    2555 scope.go:117] "RemoveContainer" containerID="87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300"
Jul 16 22:46:14.136905 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:14.136843    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm_odh-ai-gateway-infra(6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9"
Jul 16 22:46:15.144150 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:15.144121    2555 scope.go:117] "RemoveContainer" containerID="87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300"
Jul 16 22:46:15.144638 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:15.144386    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm_odh-ai-gateway-infra(6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9"
Jul 16 22:46:18.671194 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:18.671162    2555 scope.go:117] "RemoveContainer" containerID="d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc"
Jul 16 22:46:19.168352 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:19.168319    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerStarted","Data":"2f7defb559e0bae759c136a0671a3c79d1ef354420aee097640d53b35d16e483"}
Jul 16 22:46:19.184998 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:19.184898    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" podStartSLOduration=18.184882475 podStartE2EDuration="18.184882475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:19.182680645 +0000 UTC m=+3434.019734447" watchObservedRunningTime="2026-07-16 22:46:19.184882475 +0000 UTC m=+3434.021936274"
Jul 16 22:46:20.061888 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.061853    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:20.065191 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.065175    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:20.081315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.081283    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:20.125864 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.125823    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j\" (UID: \"63272d57-eb74-45c7-bad4-72210da276f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:20.175728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.175688    2555 generic.go:356] "Generic (PLEG): container finished" podID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerID="2f7defb559e0bae759c136a0671a3c79d1ef354420aee097640d53b35d16e483" exitCode=6
Jul 16 22:46:20.175728 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.175725    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerDied","Data":"2f7defb559e0bae759c136a0671a3c79d1ef354420aee097640d53b35d16e483"}
Jul 16 22:46:20.175984 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.175779    2555 scope.go:117] "RemoveContainer" containerID="d9a619c84e5b5b51f0f1b860792e049f6056dab9775532b6d0507c8b11c30bcc"
Jul 16 22:46:20.176107 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.176082    2555 scope.go:117] "RemoveContainer" containerID="2f7defb559e0bae759c136a0671a3c79d1ef354420aee097640d53b35d16e483"
Jul 16 22:46:20.176366 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:20.176348    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-gg7wv_odh-ai-gateway-infra(3cc4acfc-dd60-4c97-8a8f-211aca027317)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317"
Jul 16 22:46:20.194614 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.194580    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:20.227202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.227166    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j\" (UID: \"63272d57-eb74-45c7-bad4-72210da276f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:20.227869 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.227845    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j\" (UID: \"63272d57-eb74-45c7-bad4-72210da276f4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:20.374810 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.374707    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:20.514171 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:20.514146    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:20.515811 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:46:20.515783    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod63272d57_eb74_45c7_bad4_72210da276f4.slice/crio-638fa7437b48cd9d5fc92d40e14c66c6ab0341876cecf8e51e4b95ca00687d90 WatchSource:0}: Error finding container 638fa7437b48cd9d5fc92d40e14c66c6ab0341876cecf8e51e4b95ca00687d90: Status 404 returned error can't find the container with id 638fa7437b48cd9d5fc92d40e14c66c6ab0341876cecf8e51e4b95ca00687d90
Jul 16 22:46:21.182807 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.182772    2555 generic.go:356] "Generic (PLEG): container finished" podID="63272d57-eb74-45c7-bad4-72210da276f4" containerID="27cb8750363178415db11a602deffd4fbfaf71dfaa3c13e9c9b7130083d3fef3" exitCode=6
Jul 16 22:46:21.183243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.182832    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerDied","Data":"27cb8750363178415db11a602deffd4fbfaf71dfaa3c13e9c9b7130083d3fef3"}
Jul 16 22:46:21.183243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.182909    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerStarted","Data":"638fa7437b48cd9d5fc92d40e14c66c6ab0341876cecf8e51e4b95ca00687d90"}
Jul 16 22:46:21.183243 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.183131    2555 scope.go:117] "RemoveContainer" containerID="27cb8750363178415db11a602deffd4fbfaf71dfaa3c13e9c9b7130083d3fef3"
Jul 16 22:46:21.308375 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.308349    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:21.438327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.438287    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca\") pod \"3cc4acfc-dd60-4c97-8a8f-211aca027317\" (UID: \"3cc4acfc-dd60-4c97-8a8f-211aca027317\") "
Jul 16 22:46:21.438676 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.438654    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3cc4acfc-dd60-4c97-8a8f-211aca027317" (UID: "3cc4acfc-dd60-4c97-8a8f-211aca027317"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:21.539334 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:21.539309    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3cc4acfc-dd60-4c97-8a8f-211aca027317-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:46:22.191676 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.191642    2555 generic.go:356] "Generic (PLEG): container finished" podID="63272d57-eb74-45c7-bad4-72210da276f4" containerID="4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311" exitCode=6
Jul 16 22:46:22.192165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.191733    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerDied","Data":"4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311"}
Jul 16 22:46:22.192165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.191802    2555 scope.go:117] "RemoveContainer" containerID="27cb8750363178415db11a602deffd4fbfaf71dfaa3c13e9c9b7130083d3fef3"
Jul 16 22:46:22.192165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.191943    2555 scope.go:117] "RemoveContainer" containerID="4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311"
Jul 16 22:46:22.192348 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:22.192197    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j_odh-ai-gateway-infra(63272d57-eb74-45c7-bad4-72210da276f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" podUID="63272d57-eb74-45c7-bad4-72210da276f4"
Jul 16 22:46:22.193197 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.193184    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"
Jul 16 22:46:22.193268 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.193232    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv" event={"ID":"3cc4acfc-dd60-4c97-8a8f-211aca027317","Type":"ContainerDied","Data":"a9477115e28ce1df36b2e3c6b05485280125a69cbd923b042487de932f2df5a9"}
Jul 16 22:46:22.203161 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.203143    2555 scope.go:117] "RemoveContainer" containerID="2f7defb559e0bae759c136a0671a3c79d1ef354420aee097640d53b35d16e483"
Jul 16 22:46:22.225077 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.225052    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:22.231252 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:22.231223    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-gg7wv"]
Jul 16 22:46:23.201293 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:23.201263    2555 scope.go:117] "RemoveContainer" containerID="4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311"
Jul 16 22:46:23.201808 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:23.201504    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j_odh-ai-gateway-infra(63272d57-eb74-45c7-bad4-72210da276f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" podUID="63272d57-eb74-45c7-bad4-72210da276f4"
Jul 16 22:46:23.494495 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:23.494403    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cf7dfc9c-zstcg_848b39a2-b37d-4a14-bfb8-b250294e70bf/copy-manifests/0.log"
Jul 16 22:46:23.513689 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:23.513646    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cf7dfc9c-zstcg_848b39a2-b37d-4a14-bfb8-b250294e70bf/manager/0.log"
Jul 16 22:46:23.675999 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:23.675960    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" path="/var/lib/kubelet/pods/3cc4acfc-dd60-4c97-8a8f-211aca027317/volumes"
Jul 16 22:46:24.278624 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.278594    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29737365-g29md_5aa46f7d-5485-4a79-9537-7715257ca5f0/cleanup/0.log"
Jul 16 22:46:24.397851 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.397822    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm_6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9/revoke-keys/1.log"
Jul 16 22:46:24.516378 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.516331    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-a-14b5b4-fdt27_ead0518e-a662-4ad4-b5d5-d50aad9da903/revoke-keys/0.log"
Jul 16 22:46:24.635250 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.635153    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-ea0330-mgjm7_2976694d-2c26-470d-8447-51eada901341/revoke-keys/0.log"
Jul 16 22:46:24.870499 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.870453    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-a-fb6c08-9g2jb_004d3af7-5f6e-49de-ae4f-f881af25ca77/revoke-keys/0.log"
Jul 16 22:46:24.988209 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:24.988116    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-2dbd48-8crz5_bf964e33-aa00-4916-b5e7-2f093c03165b/revoke-keys/0.log"
Jul 16 22:46:25.102246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:25.102221    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j_63272d57-eb74-45c7-bad4-72210da276f4/revoke-keys/1.log"
Jul 16 22:46:25.217408 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:25.217374    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-5d6c742e-wh47r_262ec790-b8fb-4fe8-972f-2e3efce9fa22/revoke-keys/0.log"
Jul 16 22:46:25.336859 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:25.336832    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-f5e7ed86-4pdvq_e5dd0a35-fdfd-44f3-92c5-a0e699e2e495/revoke-keys/0.log"
Jul 16 22:46:25.567509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:25.567453    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-dbc614-zs8zm_6e193ba2-37ff-4d59-acc6-78b13e2f64a7/revoke-keys/0.log"
Jul 16 22:46:26.513315 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.513284    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/util/0.log"
Jul 16 22:46:26.521321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.521295    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/pull/0.log"
Jul 16 22:46:26.528321 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.528295    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/extract/0.log"
Jul 16 22:46:26.647123 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.647097    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/pull/0.log"
Jul 16 22:46:26.654735 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.654715    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/extract/0.log"
Jul 16 22:46:26.662229 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.662207    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/util/0.log"
Jul 16 22:46:26.779010 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.778985    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/util/0.log"
Jul 16 22:46:26.788884 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.788863    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/pull/0.log"
Jul 16 22:46:26.795977 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.795953    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/extract/0.log"
Jul 16 22:46:26.906850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.906816    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/util/0.log"
Jul 16 22:46:26.915075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.915054    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/pull/0.log"
Jul 16 22:46:26.922428 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:26.922400    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/extract/0.log"
Jul 16 22:46:27.786661 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:27.786633    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xl5tk_71e0fe90-a227-40bf-881d-60a8a40bf654/limitador/0.log"
Jul 16 22:46:28.434694 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:28.434665    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94knq_76185561-cce1-4dbe-b589-848a085a0df6/discovery/0.log"
Jul 16 22:46:28.670833 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:28.670805    2555 scope.go:117] "RemoveContainer" containerID="87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300"
Jul 16 22:46:29.237346 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:29.237313    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerStarted","Data":"5ad51884ba78bb708f369fe8dbc642efea562b64d14985d9e86acd80509b28c7"}
Jul 16 22:46:29.811198 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:29.811172    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-dsrk2_9946713d-97f7-42b8-b56b-92283a1ecc0e/main/0.log"
Jul 16 22:46:29.932989 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:29.932957    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-6b86dbgfnmk_252c0036-d381-4c71-8450-e1b066cef4a4/main/0.log"
Jul 16 22:46:30.244721 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:30.244625    2555 generic.go:356] "Generic (PLEG): container finished" podID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerID="5ad51884ba78bb708f369fe8dbc642efea562b64d14985d9e86acd80509b28c7" exitCode=6
Jul 16 22:46:30.244721 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:30.244703    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerDied","Data":"5ad51884ba78bb708f369fe8dbc642efea562b64d14985d9e86acd80509b28c7"}
Jul 16 22:46:30.245239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:30.244764    2555 scope.go:117] "RemoveContainer" containerID="87a44df826c9e7babcf91eb3799402b9942a89defae4cafc8cfb160b6593a300"
Jul 16 22:46:30.245239 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:30.245095    2555 scope.go:117] "RemoveContainer" containerID="5ad51884ba78bb708f369fe8dbc642efea562b64d14985d9e86acd80509b28c7"
Jul 16 22:46:30.245350 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:30.245328    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm_odh-ai-gateway-infra(6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9"
Jul 16 22:46:30.269654 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:30.269630    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:31.377348 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:31.377319    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:31.529927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:31.529894    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca\") pod \"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9\" (UID: \"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9\") "
Jul 16 22:46:31.530281 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:31.530256    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" (UID: "6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:31.630437 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:31.630398    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:46:32.258589 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:32.258556    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"
Jul 16 22:46:32.258834 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:32.258591    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm" event={"ID":"6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9","Type":"ContainerDied","Data":"174bd6e42914e00b734ee6a55aa7fa66c2ea283bf08c09a49e257141cceda518"}
Jul 16 22:46:32.258834 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:32.258655    2555 scope.go:117] "RemoveContainer" containerID="5ad51884ba78bb708f369fe8dbc642efea562b64d14985d9e86acd80509b28c7"
Jul 16 22:46:32.284050 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:32.284020    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:32.295073 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:32.295051    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-d3f2b144-kklqm"]
Jul 16 22:46:33.675850 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:33.675816    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" path="/var/lib/kubelet/pods/6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9/volumes"
Jul 16 22:46:34.670783 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:34.670750    2555 scope.go:117] "RemoveContainer" containerID="4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311"
Jul 16 22:46:35.277600 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:35.277570    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerStarted","Data":"daa93f80d4be9fd9eaaa7a35d8c96cf8ac50f0304e2544d573f392607e008545"}
Jul 16 22:46:35.299327 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:35.299274    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" podStartSLOduration=15.299260029 podStartE2EDuration="15.299260029s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:46:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:35.296821858 +0000 UTC m=+3450.133875657" watchObservedRunningTime="2026-07-16 22:46:35.299260029 +0000 UTC m=+3450.136313828"
Jul 16 22:46:36.285032 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:36.284998    2555 generic.go:356] "Generic (PLEG): container finished" podID="63272d57-eb74-45c7-bad4-72210da276f4" containerID="daa93f80d4be9fd9eaaa7a35d8c96cf8ac50f0304e2544d573f392607e008545" exitCode=6
Jul 16 22:46:36.285483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:36.285066    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerDied","Data":"daa93f80d4be9fd9eaaa7a35d8c96cf8ac50f0304e2544d573f392607e008545"}
Jul 16 22:46:36.285483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:36.285128    2555 scope.go:117] "RemoveContainer" containerID="4e266dac4ea7b0ec7b8239e10bf299984e78948f3dab4533f3e97ad612c7b311"
Jul 16 22:46:36.285483 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:36.285366    2555 scope.go:117] "RemoveContainer" containerID="daa93f80d4be9fd9eaaa7a35d8c96cf8ac50f0304e2544d573f392607e008545"
Jul 16 22:46:36.285659 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:46:36.285598    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 20s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j_odh-ai-gateway-infra(63272d57-eb74-45c7-bad4-72210da276f4)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" podUID="63272d57-eb74-45c7-bad4-72210da276f4"
Jul 16 22:46:36.326202 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:36.326160    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:37.431382 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:37.431361    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:37.581075 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:37.580984    2555 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca\") pod \"63272d57-eb74-45c7-bad4-72210da276f4\" (UID: \"63272d57-eb74-45c7-bad4-72210da276f4\") "
Jul 16 22:46:37.581386 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:37.581367    2555 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "63272d57-eb74-45c7-bad4-72210da276f4" (UID: "63272d57-eb74-45c7-bad4-72210da276f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:46:37.682029 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:37.682002    2555 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/63272d57-eb74-45c7-bad4-72210da276f4-maas-api-service-ca\") on node \"ip-10-0-131-79.ec2.internal\" DevicePath \"\""
Jul 16 22:46:38.299523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.299493    2555 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"
Jul 16 22:46:38.299706 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.299519    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j" event={"ID":"63272d57-eb74-45c7-bad4-72210da276f4","Type":"ContainerDied","Data":"638fa7437b48cd9d5fc92d40e14c66c6ab0341876cecf8e51e4b95ca00687d90"}
Jul 16 22:46:38.299706 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.299585    2555 scope.go:117] "RemoveContainer" containerID="daa93f80d4be9fd9eaaa7a35d8c96cf8ac50f0304e2544d573f392607e008545"
Jul 16 22:46:38.338501 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.338456    2555 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:38.350993 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.350966    2555 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-39074f9f-2rk2j"]
Jul 16 22:46:38.875655 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:38.875621    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-jwnj9_f98e5fc8-14b8-4608-a52f-8fc5efaf8778/global-pull-secret-syncer/0.log"
Jul 16 22:46:39.317657 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:39.317626    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pvxqn_4e959648-65d9-45ea-ac05-3522b5cc0041/konnectivity-agent/0.log"
Jul 16 22:46:39.422863 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:39.422833    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-79.ec2.internal_e8d0181de62365719ca8addca6e9fe43/haproxy/0.log"
Jul 16 22:46:39.675898 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:39.675816    2555 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="63272d57-eb74-45c7-bad4-72210da276f4" path="/var/lib/kubelet/pods/63272d57-eb74-45c7-bad4-72210da276f4/volumes"
Jul 16 22:46:43.381923 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.381885    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/extract/0.log"
Jul 16 22:46:43.414031 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.414000    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/util/0.log"
Jul 16 22:46:43.448153 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.448127    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_0acee64185f523d1d1272e9af2e4d9333e0dcde792ba30e1fa9605b7596r278_b91169da-c866-4383-b08f-2dc5b7d52914/pull/0.log"
Jul 16 22:46:43.479251 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.479224    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/extract/0.log"
Jul 16 22:46:43.513655 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.513626    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/util/0.log"
Jul 16 22:46:43.542227 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.542199    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_19cb86e64775c5699d5aacf881a09c2d51e7f55f9e1ff096f2a667c5e0b6czm_e932528d-0c36-4366-828e-7b9a029bdfab/pull/0.log"
Jul 16 22:46:43.573584 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.573554    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/extract/0.log"
Jul 16 22:46:43.604696 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.604669    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/util/0.log"
Jul 16 22:46:43.633797 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.633713    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_5fc36eb1065777914bfe0ff85f9a202e2a3bafefb563d9e2994474ed73kwlkk_02182b5c-0f36-4b77-945c-cc1c8c36305c/pull/0.log"
Jul 16 22:46:43.673957 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.673921    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/extract/0.log"
Jul 16 22:46:43.702725 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.702700    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/util/0.log"
Jul 16 22:46:43.740569 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:43.740546    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_9438e18bbad664a016701a3153a6b421e2d977be7eb0117f80bcf45ef1v29s7_6fbcdc05-f3f0-4335-bc0a-57fc8b55ec13/pull/0.log"
Jul 16 22:46:44.429164 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:44.429131    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-xl5tk_71e0fe90-a227-40bf-881d-60a8a40bf654/limitador/0.log"
Jul 16 22:46:46.412203 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.412106    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/alertmanager/0.log"
Jul 16 22:46:46.446767 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.446743    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/config-reloader/0.log"
Jul 16 22:46:46.477952 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.477926    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/kube-rbac-proxy-web/0.log"
Jul 16 22:46:46.512142 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.512104    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/kube-rbac-proxy/0.log"
Jul 16 22:46:46.544443 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.544414    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/kube-rbac-proxy-metric/0.log"
Jul 16 22:46:46.571788 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.571760    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/prom-label-proxy/0.log"
Jul 16 22:46:46.602003 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.601978    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_6dd0018c-c738-473e-bca1-670a807161b1/init-config-reloader/0.log"
Jul 16 22:46:46.752370 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.752289    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lm4pt_190d05e8-b28f-4be5-a490-3b90c3b68f1f/kube-state-metrics/0.log"
Jul 16 22:46:46.903390 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.903363    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lm4pt_190d05e8-b28f-4be5-a490-3b90c3b68f1f/kube-rbac-proxy-main/0.log"
Jul 16 22:46:46.965670 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:46.965640    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-lm4pt_190d05e8-b28f-4be5-a490-3b90c3b68f1f/kube-rbac-proxy-self/0.log"
Jul 16 22:46:47.075982 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.075948    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-q7bb2_97b12f50-174c-42ec-b704-f54055e78d09/monitoring-plugin/0.log"
Jul 16 22:46:47.126631 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.126604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw5qf_27c4e46d-1427-4983-86e5-f61c1601c9a8/node-exporter/0.log"
Jul 16 22:46:47.161049 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.161019    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw5qf_27c4e46d-1427-4983-86e5-f61c1601c9a8/kube-rbac-proxy/0.log"
Jul 16 22:46:47.216966 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.216937    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bw5qf_27c4e46d-1427-4983-86e5-f61c1601c9a8/init-textfile/0.log"
Jul 16 22:46:47.772509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.772407    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7kx98_ef203e7b-67b3-42d9-82be-9f4dbb20a4e6/prometheus-operator/0.log"
Jul 16 22:46:47.794724 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.794686    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7kx98_ef203e7b-67b3-42d9-82be-9f4dbb20a4e6/kube-rbac-proxy/0.log"
Jul 16 22:46:47.820878 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.820850    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-l9jsr_b57c7049-ec5e-49c7-84e8-b69caa8ed260/prometheus-operator-admission-webhook/0.log"
Jul 16 22:46:47.851927 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.851898    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767f5f56bc-2d8qn_22b26901-2951-410a-9867-058784aae2d9/telemeter-client/0.log"
Jul 16 22:46:47.873381 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.873357    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767f5f56bc-2d8qn_22b26901-2951-410a-9867-058784aae2d9/reload/0.log"
Jul 16 22:46:47.897903 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:47.897873    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-767f5f56bc-2d8qn_22b26901-2951-410a-9867-058784aae2d9/kube-rbac-proxy/0.log"
Jul 16 22:46:48.824018 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.823965    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"]
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824382    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824397    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824410    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824415    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824421    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824427    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824438 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824440    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824445    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824475    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824480    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824489    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824497    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824558    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824565    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824572    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824578    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824584    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824592    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:46:48.824727 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.824596    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:46:48.827632 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.827616    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.830256 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.830233    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zq9hl\"/\"openshift-service-ca.crt\""
Jul 16 22:46:48.831182 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.831164    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-zq9hl\"/\"default-dockercfg-q7xtq\""
Jul 16 22:46:48.831311 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.831169    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-zq9hl\"/\"kube-root-ca.crt\""
Jul 16 22:46:48.836789 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.836762    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"]
Jul 16 22:46:48.871640 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.871617    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbp9r\" (UniqueName: \"kubernetes.io/projected/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-kube-api-access-cbp9r\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.871782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.871694    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-sys\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.871782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.871731    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-proc\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.871944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.871789    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-lib-modules\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.871944 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.871827    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-podres\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972187 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972145    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbp9r\" (UniqueName: \"kubernetes.io/projected/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-kube-api-access-cbp9r\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972216    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-sys\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972253    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-proc\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972292    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-lib-modules\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972360 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972331    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-podres\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972331    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-sys\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972412    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-proc\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972503    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-lib-modules\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.972612 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.972504    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-podres\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:48.981971 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:48.981952    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbp9r\" (UniqueName: \"kubernetes.io/projected/3e8f3001-45c7-4e67-bbab-06f5fd5bbc10-kube-api-access-cbp9r\") pod \"perf-node-gather-daemonset-vbhxg\" (UID: \"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10\") " pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:49.138534 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:49.138441    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:49.277343 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:49.277318    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"]
Jul 16 22:46:49.278943 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:46:49.278915    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod3e8f3001_45c7_4e67_bbab_06f5fd5bbc10.slice/crio-d2223a9aef84e1189d4b79d71c4cd81513af3dff40330ce5824715349155f940 WatchSource:0}: Error finding container d2223a9aef84e1189d4b79d71c4cd81513af3dff40330ce5824715349155f940: Status 404 returned error can't find the container with id d2223a9aef84e1189d4b79d71c4cd81513af3dff40330ce5824715349155f940
Jul 16 22:46:49.363437 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:49.363410    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg" event={"ID":"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10","Type":"ContainerStarted","Data":"d2223a9aef84e1189d4b79d71c4cd81513af3dff40330ce5824715349155f940"}
Jul 16 22:46:50.370144 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:50.370096    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg" event={"ID":"3e8f3001-45c7-4e67-bbab-06f5fd5bbc10","Type":"ContainerStarted","Data":"b6abd2d2576e428fbe27525e8b4fd38dca4c65346f8cccc0cfcfb877dbb278c5"}
Jul 16 22:46:50.370544 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:50.370247    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:50.391246 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:50.391198    2555 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg" podStartSLOduration=2.391184526 podStartE2EDuration="2.391184526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:46:50.38833928 +0000 UTC m=+3465.225393104" watchObservedRunningTime="2026-07-16 22:46:50.391184526 +0000 UTC m=+3465.228238325"
Jul 16 22:46:50.622544 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:50.622449    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-wvl26_b89ccd51-d8d9-439e-9219-8a1e30821fce/download-server/0.log"
Jul 16 22:46:52.656082 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:52.656054    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmwpc_12e3e88b-17d1-43ae-82ea-b84c3bb16fd7/dns/0.log"
Jul 16 22:46:52.719856 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:52.719827    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-bmwpc_12e3e88b-17d1-43ae-82ea-b84c3bb16fd7/kube-rbac-proxy/0.log"
Jul 16 22:46:52.820798 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:52.820766    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gbvhz_80d51c90-35f7-4d89-8ef7-ef6667c84f03/dns-node-resolver/0.log"
Jul 16 22:46:53.465289 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:53.465258    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-d2cf5_df0d312c-dd41-4069-bd73-e9fc466f8544/node-ca/0.log"
Jul 16 22:46:54.671965 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:54.671932    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94knq_76185561-cce1-4dbe-b589-848a085a0df6/discovery/0.log"
Jul 16 22:46:55.650008 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:55.649981    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zq5f8_65bc0ec8-d3fa-474b-a8c7-ea4dd10c5b07/serve-healthcheck-canary/0.log"
Jul 16 22:46:56.256582 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:56.256551    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czpz4_7e4804b0-0f3e-45f5-87ea-6d4f28651a09/kube-rbac-proxy/0.log"
Jul 16 22:46:56.281628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:56.281604    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czpz4_7e4804b0-0f3e-45f5-87ea-6d4f28651a09/exporter/0.log"
Jul 16 22:46:56.303667 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:56.303637    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-czpz4_7e4804b0-0f3e-45f5-87ea-6d4f28651a09/extractor/0.log"
Jul 16 22:46:56.386312 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:56.386287    2555 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-zq9hl/perf-node-gather-daemonset-vbhxg"
Jul 16 22:46:59.399337 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:59.399301    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cf7dfc9c-zstcg_848b39a2-b37d-4a14-bfb8-b250294e70bf/manager/0.log"
Jul 16 22:46:59.418077 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:46:59.418055    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-54cf7dfc9c-zstcg_848b39a2-b37d-4a14-bfb8-b250294e70bf/copy-manifests/0.log"
Jul 16 22:47:00.163915 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.163819    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"]
Jul 16 22:47:00.164292 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164272    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:47:00.164391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164296    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:47:00.164391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164309    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:47:00.164391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164317    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="63272d57-eb74-45c7-bad4-72210da276f4" containerName="revoke-keys"
Jul 16 22:47:00.164391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164335    2555 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:47:00.164391 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164344    2555 state_mem.go:107] "Deleted CPUSet assignment" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:47:00.164685 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164428    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cc4acfc-dd60-4c97-8a8f-211aca027317" containerName="revoke-keys"
Jul 16 22:47:00.164685 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.164444    2555 memory_manager.go:368] "RemoveStaleState removing state" podUID="6059a2a0-19c2-4f5f-b2a8-83e2acbf8ec9" containerName="revoke-keys"
Jul 16 22:47:00.168109 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.168050    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"
Jul 16 22:47:00.170935 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.170911    2555 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-gknhg\""
Jul 16 22:47:00.171058 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.170904    2555 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 16 22:47:00.178057 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.177607    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"]
Jul 16 22:47:00.230773 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.230738    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94knq_76185561-cce1-4dbe-b589-848a085a0df6/discovery/0.log"
Jul 16 22:47:00.268240 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.268202    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14840ff2-8749-442d-aaef-4be2e2b00a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t\" (UID: \"14840ff2-8749-442d-aaef-4be2e2b00a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"
Jul 16 22:47:00.369221 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.369183    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14840ff2-8749-442d-aaef-4be2e2b00a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t\" (UID: \"14840ff2-8749-442d-aaef-4be2e2b00a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"
Jul 16 22:47:00.369835 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.369818    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/14840ff2-8749-442d-aaef-4be2e2b00a5d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t\" (UID: \"14840ff2-8749-442d-aaef-4be2e2b00a5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"
Jul 16 22:47:00.483042 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.482956    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"
Jul 16 22:47:00.637165 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.637144    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t"]
Jul 16 22:47:00.638425 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:47:00.638396    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14840ff2_8749_442d_aaef_4be2e2b00a5d.slice/crio-5d6edc720030f83d52f0ab88426fd8712455aeaa1a21384b24e4de966ef68e0d WatchSource:0}: Error finding container 5d6edc720030f83d52f0ab88426fd8712455aeaa1a21384b24e4de966ef68e0d: Status 404 returned error can't find the container with id 5d6edc720030f83d52f0ab88426fd8712455aeaa1a21384b24e4de966ef68e0d
Jul 16 22:47:00.715439 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.715412    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-656bfb4b98-6gjq4_a79e15e6-e1c5-410b-b171-535033d810ea/manager/0.log"
Jul 16 22:47:00.793876 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:00.793843    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-8zc4c_c8f7e5ff-13b7-41da-9e1b-f7d1b0bb6098/openshift-lws-operator/0.log"
Jul 16 22:47:01.433333 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.433297    2555 generic.go:356] "Generic (PLEG): container finished" podID="14840ff2-8749-442d-aaef-4be2e2b00a5d" containerID="5c1b234748944c554a7920a97e74a14c000c8e7c68ca2df1fe28ac451a0c5ebe" exitCode=6
Jul 16 22:47:01.433523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.433376    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t" event={"ID":"14840ff2-8749-442d-aaef-4be2e2b00a5d","Type":"ContainerDied","Data":"5c1b234748944c554a7920a97e74a14c000c8e7c68ca2df1fe28ac451a0c5ebe"}
Jul 16 22:47:01.433523 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.433433    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t" event={"ID":"14840ff2-8749-442d-aaef-4be2e2b00a5d","Type":"ContainerStarted","Data":"5d6edc720030f83d52f0ab88426fd8712455aeaa1a21384b24e4de966ef68e0d"}
Jul 16 22:47:01.433618 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.433605    2555 scope.go:117] "RemoveContainer" containerID="5c1b234748944c554a7920a97e74a14c000c8e7c68ca2df1fe28ac451a0c5ebe"
Jul 16 22:47:01.884893 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.884858    2555 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-94bb8fbfd-94knq_76185561-cce1-4dbe-b589-848a085a0df6/discovery/0.log"
Jul 16 22:47:01.917551 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.917521    2555 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"]
Jul 16 22:47:01.922527 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.922509    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"
Jul 16 22:47:01.957865 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.957825    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"]
Jul 16 22:47:01.984509 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:01.984430    2555 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525be039-a108-4335-b842-7911c18f413e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-p9vgs\" (UID: \"525be039-a108-4335-b842-7911c18f413e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"
Jul 16 22:47:02.085694 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.085659    2555 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525be039-a108-4335-b842-7911c18f413e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-p9vgs\" (UID: \"525be039-a108-4335-b842-7911c18f413e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"
Jul 16 22:47:02.086328 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.086303    2555 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/525be039-a108-4335-b842-7911c18f413e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-23b64857-p9vgs\" (UID: \"525be039-a108-4335-b842-7911c18f413e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"
Jul 16 22:47:02.232596 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.232553    2555 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"
Jul 16 22:47:02.398626 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.398591    2555 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs"]
Jul 16 22:47:02.402240 ip-10-0-131-79 kubenswrapper[2555]: W0716 22:47:02.402184    2555 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod525be039_a108_4335_b842_7911c18f413e.slice/crio-c6918ea8d293dce297135e693d7320e5e2942712ad90f42f729cd5cf15648802 WatchSource:0}: Error finding container c6918ea8d293dce297135e693d7320e5e2942712ad90f42f729cd5cf15648802: Status 404 returned error can't find the container with id c6918ea8d293dce297135e693d7320e5e2942712ad90f42f729cd5cf15648802
Jul 16 22:47:02.440628 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.440599    2555 generic.go:356] "Generic (PLEG): container finished" podID="14840ff2-8749-442d-aaef-4be2e2b00a5d" containerID="6bc963f0c1c42e8a99269973ca738853c09e4456b5281751ad1b09d1c66d4671" exitCode=6
Jul 16 22:47:02.440754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.440685    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t" event={"ID":"14840ff2-8749-442d-aaef-4be2e2b00a5d","Type":"ContainerDied","Data":"6bc963f0c1c42e8a99269973ca738853c09e4456b5281751ad1b09d1c66d4671"}
Jul 16 22:47:02.440754 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.440741    2555 scope.go:117] "RemoveContainer" containerID="5c1b234748944c554a7920a97e74a14c000c8e7c68ca2df1fe28ac451a0c5ebe"
Jul 16 22:47:02.440928 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.440910    2555 scope.go:117] "RemoveContainer" containerID="6bc963f0c1c42e8a99269973ca738853c09e4456b5281751ad1b09d1c66d4671"
Jul 16 22:47:02.441154 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:47:02.441134    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t_odh-ai-gateway-infra(14840ff2-8749-442d-aaef-4be2e2b00a5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t" podUID="14840ff2-8749-442d-aaef-4be2e2b00a5d"
Jul 16 22:47:02.442030 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:02.442010    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs" event={"ID":"525be039-a108-4335-b842-7911c18f413e","Type":"ContainerStarted","Data":"c6918ea8d293dce297135e693d7320e5e2942712ad90f42f729cd5cf15648802"}
Jul 16 22:47:03.449331 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:03.449300    2555 scope.go:117] "RemoveContainer" containerID="6bc963f0c1c42e8a99269973ca738853c09e4456b5281751ad1b09d1c66d4671"
Jul 16 22:47:03.449753 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:47:03.449552    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t_odh-ai-gateway-infra(14840ff2-8749-442d-aaef-4be2e2b00a5d)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-f757c4eb-jpk7t" podUID="14840ff2-8749-442d-aaef-4be2e2b00a5d"
Jul 16 22:47:03.450519 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:03.450502    2555 generic.go:356] "Generic (PLEG): container finished" podID="525be039-a108-4335-b842-7911c18f413e" containerID="cdd53825ca7763fb84967cadd01230033cc19be6916ac693df2dc56af91d17ae" exitCode=6
Jul 16 22:47:03.450601 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:03.450581    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs" event={"ID":"525be039-a108-4335-b842-7911c18f413e","Type":"ContainerDied","Data":"cdd53825ca7763fb84967cadd01230033cc19be6916ac693df2dc56af91d17ae"}
Jul 16 22:47:03.450707 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:03.450694    2555 scope.go:117] "RemoveContainer" containerID="cdd53825ca7763fb84967cadd01230033cc19be6916ac693df2dc56af91d17ae"
Jul 16 22:47:04.458320 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:04.458290    2555 generic.go:356] "Generic (PLEG): container finished" podID="525be039-a108-4335-b842-7911c18f413e" containerID="f671cc3dfc3dbcfdd259adacae6c465cc192f3043d6012541f228d2bb528cacf" exitCode=6
Jul 16 22:47:04.458782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:04.458388    2555 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs" event={"ID":"525be039-a108-4335-b842-7911c18f413e","Type":"ContainerDied","Data":"f671cc3dfc3dbcfdd259adacae6c465cc192f3043d6012541f228d2bb528cacf"}
Jul 16 22:47:04.458782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:04.458444    2555 scope.go:117] "RemoveContainer" containerID="cdd53825ca7763fb84967cadd01230033cc19be6916ac693df2dc56af91d17ae"
Jul 16 22:47:04.458782 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:04.458592    2555 scope.go:117] "RemoveContainer" containerID="f671cc3dfc3dbcfdd259adacae6c465cc192f3043d6012541f228d2bb528cacf"
Jul 16 22:47:04.458918 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:47:04.458812    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-p9vgs_odh-ai-gateway-infra(525be039-a108-4335-b842-7911c18f413e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs" podUID="525be039-a108-4335-b842-7911c18f413e"
Jul 16 22:47:05.465900 ip-10-0-131-79 kubenswrapper[2555]: I0716 22:47:05.465813    2555 scope.go:117] "RemoveContainer" containerID="f671cc3dfc3dbcfdd259adacae6c465cc192f3043d6012541f228d2bb528cacf"
Jul 16 22:47:05.466367 ip-10-0-131-79 kubenswrapper[2555]: E0716 22:47:05.466007    2555 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-23b64857-p9vgs_odh-ai-gateway-infra(525be039-a108-4335-b842-7911c18f413e)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-23b64857-p9vgs" podUID="525be039-a108-4335-b842-7911c18f413e"